Policy optimization method and device using environmental model based on memristor array

Through the dynamic environment model and policy gradient optimization algorithm based on memristor array, the uncertainty problem of traditional neural networks in long-term task planning in dynamic systems is solved, and stable policy optimization and efficient data utilization are achieved.

CN114819093BActive Publication Date: 2024-11-01TSINGHUA UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210497721.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-09
Publication Date
2024-11-01
Estimated Expiration
2042-05-09

AI Technical Summary

Technical Problem

Traditional artificial neural networks lack the ability to model uncertainty in the long-term task planning of dynamic systems, resulting in estimation deviation from actual behavior, and inefficient data in deep reinforcement learning, making it impossible to effectively optimize strategies.

Method used

Using a dynamic environment model based on memristor array, the state and hidden input variables of the dynamic system are obtained through Bayesian neural network, and a strategic gradient optimization algorithm such as REINFORCE, PRO or TPRO algorithm is used to generate an optimization cost data sample set and optimize object strategies. .

Benefits of technology

It effectively solves the problems of gradient disappearance and explosion, improves the stability and data efficiency of strategy search, and can be planned and optimized in a dynamic environment for a long time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114819093B_ABST
    Figure CN114819093B_ABST
Patent Text Reader

Abstract

A policy optimization method and a policy optimization device using a dynamic environment model based on a memristor array. The method includes: obtaining a dynamic environment model based on a memristor array; making multiple predictions at multiple moments according to the dynamic environment model and the object strategy to obtain a data sample set including the optimization cost of the object strategy corresponding to the multiple moments; based on the data sample set, using a policy gradient optimization algorithm to perform policy search to optimize the object strategy. The method uses a dynamic environment model based on a memristor array to generate a data sample set, realizes long-term dynamic programming based on the dynamic environment model, and then uses a more stable algorithm such as a policy gradient optimization algorithm to perform policy search, which can effectively optimize the object strategy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a strategy optimization method and a strategy optimization device using a dynamic environment model based on a memristor array. Background Art

[0002] Artificial Neural Networks (ANNs) are widely used in modeling dynamic systems. However, long-term mission planning using traditional ANNs remains a challenge due to the lack of ability to model uncertainty. The inherent randomness (uncertainty) of real systems - process noise and approximation errors introduced by data-driven modeling can cause the long-term estimates of ANNs to deviate from the actual behavior of the system. Probabilistic models provide a way to address uncertainty. These models enable people to make informed decisions using the model's predictions while being cautious about the uncertainty of these predictions. Summary of the invention

[0003] At least one embodiment of the present disclosure provides a policy optimization method using a dynamic environment model based on a memristor array, comprising: obtaining a dynamic environment model based on a memristor array; performing multiple predictions at multiple moments according to the dynamic environment model and an object policy to obtain a data sample set including optimization costs of the object policy corresponding to the multiple moments; and performing a policy search based on the data sample set using a policy gradient optimization algorithm to optimize the object policy.

[0004] For example, in a strategy optimization method provided in an embodiment of the present disclosure, a dynamic environment model is obtained, including: obtaining a Bayesian neural network, the Bayesian neural network having a trained weight matrix; obtaining a corresponding plurality of target conductance values ​​according to the weight matrix of the Bayesian neural network, and mapping the plurality of target conductance values ​​to a memristor array; inputting the state and latent input variables corresponding to the dynamic system at time t as input signals into the weight-mapped memristor array, processing the state and latent input variables at time t according to the Bayesian neural network through the memristor array, obtaining an output signal corresponding to the processing result from the memristor array, and the output signal being used to obtain a prediction result of the dynamic system at time t+1.

[0005] For example, in the strategy optimization method provided in an embodiment of the present disclosure, the dynamic environment model is expressed as s t+1 =f(s t ,a t ; W,ε),s t is the state of the dynamic system at time t, a t is the action of the object strategy at time t, W is the weight matrix of the Bayesian neural network, ε is the additive noise corresponding to the memristor array, s t+1is the prediction result of the dynamic system at time t+1; the action a of the object strategy at time t t =π(s t ; Wπ), π represents the function of the object strategy, Wπ represents the strategy parameter, the weight matrix W of the Bayesian neural network satisfies the distribution W~q(W), and the additive noise ε is the additive Gaussian noise ε~N(0,σ 2 ).

[0006] For example, in a strategy optimization method provided in an embodiment of the present disclosure, the multiple moments include moment 1 to moment T arranged in sequence from early to late, and multiple predictions are made at the multiple moments according to the dynamic environment model and the object strategy to obtain a data sample set including the optimization cost of the object strategy corresponding to the multiple moments, including: for any moment t-1 from moment 1 to moment T, the object strategy obtains the execution action a t-1 , by a t-1 =π(s t-1 ; Wπ) obtain the action a of the target strategy at time t-1 t-1 ; According to the dynamic environment model t =f(s t-1 ,a t-1 ; W, ε) calculates the state s at the next time t after time t-1 t And obtain the state s corresponding to time t t The cost of t , thus obtaining the cost sequence {c1,c2,…,c t}, based on the cost sequence, the optimized cost J at time t is obtained t-1 , where 1≤t≤T; the data sample set from time 1 to time T is obtained {[a0,J0],…,[a T-1 ,J T-1 ]}.

[0007] For example, in the strategy optimization method provided in an embodiment of the present disclosure, the cost c at time t t The expected value of t ], then the optimization cost at time t can be obtained by To obtain.

[0008] For example, in the strategy optimization method provided in an embodiment of the present disclosure, the cost also includes cost changes caused by accidental uncertainty and cost changes caused by cognitive uncertainty, where accidental uncertainty is caused by hidden input variables and cognitive uncertainty is caused by intrinsic noise of the memristor array.

[0009] For example, in the strategy optimization method provided in an embodiment of the present disclosure, the optimization cost at time t is calculated by To obtain, σ(η,θ) is a function of aleatoric uncertainty and epistemic uncertainty, η represents aleatoric uncertainty, and θ represents epistemic uncertainty.

[0010] For example, in a strategy optimization method provided in an embodiment of the present disclosure, according to the dynamic environment model s t =f(s t-1 ,a t-1 ; W,ε) calculate the state s at time t t And obtain the state s corresponding to time t t The cost of t , including: sampling the hidden input variable z from the p(z) distribution to obtain a sample; combining the sample and the state s at time t-1 t-1 Input to the weight-mapped memristor array to get the predicted state s t ; For the predicted state s t , get the cost c t =c(s t ).

[0011] For example, in the policy optimization method provided in an embodiment of the present disclosure, the policy gradient optimization algorithm includes the REINFORCE algorithm, the PRO algorithm or the TPRO algorithm.

[0012] At least one embodiment of the present disclosure also provides a policy optimization device using a dynamic environment model based on a memristor array, including: an acquisition unit, configured to acquire a dynamic environment model based on a memristor array; a calculation unit, configured to perform multiple predictions at multiple moments according to the dynamic environment model and the object policy, and obtain a data sample set including the optimization costs of the object policy corresponding to the multiple moments; a policy search unit, configured to perform a policy search based on the data sample set using a policy gradient optimization algorithm to optimize the object policy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0014] Figure 1A A schematic flow chart of a strategy optimization method for a dynamic environment model based on a memristor array provided by at least one embodiment of the present disclosure is shown;

[0015] Figure 1B Shows Figure 1A Schematic flow chart of step S101;

[0016] Figure 2A shows a schematic structure of a memristor array;

[0017] Figure 2B is a schematic diagram of a memristor device;

[0018] Figure 2C is a schematic diagram of another memristor device;

[0019] Figure 2D A schematic diagram of mapping the weight matrix of a Bayesian neural network to a memristor array is shown;

[0020] Figure 3 Shows Figure 1A Schematic flow chart of step S102;

[0021] Figure 4 A schematic diagram showing an example of a strategy optimization method provided by at least one embodiment of the present disclosure;

[0022] Figure 5 A schematic block diagram of a strategy optimization device using a dynamic environment model based on a memristor array provided by at least one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0024] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0025] In model-free deep reinforcement learning, the agent usually needs to conduct a lot of trial and error interactions with the real environment, which is not data efficient and cannot be applied to real tasks with high trial and error costs. Model-based deep reinforcement learning can use data more efficiently. In model-based deep reinforcement learning, the agent first learns the dynamic environment model from historical experience of interacting with the real dynamic environment (such as state transition data collected in advance), and then interacts with the dynamic environment model to obtain a sub-optimal strategy.

[0026] Model-based reinforcement learning methods learn a precise dynamic environment model. When training an agent, this model is used. There is no need to interact with the real environment too many times. The agent can "imagine" the feeling of interacting in the real environment, which can greatly improve data efficiency and is applicable to actual physical scenarios where the cost of obtaining data is high. At the same time, the dynamic environment model can predict the unknown state of the environment, generalize the cognition of the agent, and can also serve as a new data source to provide contextual information to help decision-making, thereby alleviating the exploration-exploitation dilemma. When modeling the actual environment, the inherent randomness (uncertainty) of the environment - process noise and approximation errors introduced by data-driven modeling can cause the long-term estimates of artificial neural networks to deviate from the actual behavior of the system. Probabilistic models provide a way to solve the problem of uncertainty. These models enable the use of model predictions to make wise decisions while being cautious about the uncertainty of these predictions.

[0027] The inventors found that: Bayesian Neural Network (BNN) is a probabilistic model that places a neural network in a Bayesian framework, which can describe complex random patterns; and Bayesian Neural Network with latent input variables (BNN+LV) can describe complex random patterns through the distribution of latent input variables (accidental uncertainty), while considering the uncertainty of the model through the distribution of weights (epistemic uncertainty). A latent input variable is a variable that cannot be directly observed but has an impact on the state and output of the probability model. The inventors described a method and device for implementing a Bayesian neural network using intrinsic noise of a memristor in the Chinese invention patent application CN110956256A, which is cited in its entirety as part of this application.

[0028] The structure of the Bayesian neural network includes but is not limited to a fully connected structure, a convolutional neural network (CNN) structure, etc., and its network weight W is a random variable based on a certain distribution (W~q(W)).

[0029] The inventors further discovered that, assuming that there is a data set D = {X, Y} of a dynamic system for a Bayesian neural network, where X is the state feature vector of the dynamic system and Y is the next state of the dynamic system. The input of the Bayesian neural network is the state feature vector X of the dynamic system and the latent input variable z (z~p(z)); the parameters of the Bayesian neural network can be trained; and the output of the Bayesian neural network is superimposed with independent additive Gaussian noise ε (ε~N(0,σ 2 )) is to predict the next state of the dynamic system, that is, y = f(X, z, W, ε). Therefore, after the training is completed, each weight of the Bayesian neural network is a distribution. For example, each weight is an independent distribution of each other.

[0030] In long-term planning tasks, the gradient will undergo multiple steps of back propagation, resulting in gradient vanishing and explosion problems. At the same time, when a neural network directly implemented based on a memristor array performs strategy search, due to the intrinsic random characteristics of the memristor, additional noise will be introduced when the gradient is back propagated through the memristor array. These noisy gradients cannot effectively optimize strategy search.

[0031] At least one embodiment of the present disclosure provides a policy optimization method using a dynamic environment model based on a memristor array, comprising: obtaining a dynamic environment model based on a memristor array; performing multiple predictions at multiple moments according to the dynamic environment model and an object policy to obtain a data sample set including optimization costs of the object policy corresponding to the multiple moments; and performing a policy search based on the data sample set using a policy gradient optimization algorithm to optimize the object policy.

[0032] The policy optimization method provided in the above-mentioned embodiments of the present disclosure utilizes a dynamic environment model based on a memristor array to generate a data sample set, implements long-term dynamic programming based on the dynamic environment model, and then uses a more stable algorithm such as a policy gradient optimization algorithm to perform policy search, without the gradient vanishing and explosion problems, and can effectively optimize the object policy.

[0033] At least one embodiment of the present disclosure also provides a strategy optimization device corresponding to the above-mentioned strategy optimization method.

[0034] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0035] Figure 1A A schematic flow chart of a strategy optimization method for a dynamic environment model based on a memristor array provided by at least one embodiment of the present disclosure is shown.

[0036] like Figure 1A As shown, the strategy optimization method includes the following steps S101 to S103.

[0037] Step S101: obtaining a dynamic environment model based on a memristor array.

[0038] In the embodiments of the present disclosure, for example, the dynamic system can be modeled using the BNN+LV based on the memristor array to obtain a dynamic environment model. The specific steps for this are described in Figure 1B It is shown in the figure and will not be described in detail here.

[0039] Step S102: performing multiple predictions at multiple moments according to the dynamic environment model and the object strategy, and obtaining a data sample set including the optimization costs of the object strategy corresponding to the multiple moments.

[0040] For example, the object policy involved is used in deep reinforcement learning, for example, it can be the strategy of the agent to maximize the reward or achieve a specific goal during the interaction with the environment.

[0041] Step S103: Based on the data sample set, a policy gradient optimization algorithm is used to perform policy search to optimize the object policy.

[0042] For example, in different examples of the embodiments of the present disclosure, the policy gradient optimization algorithm may include the REINFORCE algorithm, the PRO (Proximal Policy Optimization) algorithm, or the TPRO (Trust Region Policy Optimization) algorithm. In the embodiments of the present disclosure, these policy gradient optimization methods are more stable and can effectively optimize the object policy.

[0043] Figure 1B Shows Figure 1A Schematic flowchart of an example of step S101 in .

[0044] like Figure 1B As shown, the example of step S101 may include the following steps S111 to S113.

[0045] Step S111: Acquire a Bayesian neural network, wherein the Bayesian neural network has a weight matrix obtained through training.

[0046] For example, the structure of the Bayesian neural network includes a fully connected structure or a convolutional neural network structure, etc. Each network weight of the Bayesian neural network is a random variable. For example, after the Bayesian neural network is trained, each weight is a distribution, such as a Gaussian distribution or a Laplace distribution.

[0047] For example, the Bayesian neural network can be trained offline to obtain a weight matrix. The method for training the Bayesian neural network can refer to conventional methods, such as using a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), etc. for training, which will not be repeated here.

[0048] Step S112: obtaining a plurality of corresponding target conductance values ​​according to the weight matrix of the Bayesian neural network, and mapping the plurality of target conductance values ​​to the memristor array.

[0049] After the Bayesian neural network training is completed to obtain the weight matrix, the weight matrix is ​​processed to obtain the corresponding multiple target conductance values. For example, in this process, the weight matrix can be biased and scaled until the weight matrix satisfies the appropriate conductance window for the memristor array used. After the weight matrix is ​​biased and scaled, the target conductance value is calculated based on the processed weight matrix and the conductance value of the memristor. The specific process of calculating the target conductance value can refer to the relevant description of the Bayesian neural network based on the memristor, which will not be repeated here.

[0050] Figure 2A The schematic structure of a memristor array is shown. The memristor array is composed of a plurality of memristor units, for example. The plurality of memristor units form an array of M rows and N columns, where M and N are both positive integers. Each memristor unit includes a switch element and one or more memristors. Figure 2A In, WL <1> , WL <2> ...WL <m>Respectively represent the word lines of the first row, the second row, ... the Mth row, and the control electrode (eg, the gate of the transistor) of the switch element in each row of the memristor unit circuit is connected to the word line corresponding to the row; BL <1> BL <2> ...BL <n>They represent the bit lines of the first column, the second column, ... the Nth column respectively, and the memristors in the memristor unit circuit of each column are connected to the bit lines corresponding to the column; SL <1> , SL <2> ...SL <m>The source lines of the first row, the second row, ... the Mth row are represented respectively, and the source of the transistor in the memristor unit circuit of each row is connected to the source line corresponding to the row. According to Kirchhoff's law, by setting the state (e.g., resistance value) of the memristor unit and applying corresponding word line signals and bit line signals to the word line and bit line, the above memristor array can complete multiplication and accumulation calculations in parallel.

[0051] Figure 2B Schematic diagram of a memristor device, the memristor device includes a memristor array and a peripheral driving circuit. Figure 2B As shown, the memristor device includes a signal acquisition device, a word line driving circuit, a bit line driving circuit, a source line driving circuit, a memristor array and a data output circuit.

[0052] For example, the signal acquisition device is configured to convert the digital signal into a plurality of analog signals through a digital to analog converter (DAC) to input the analog signals into a plurality of column signal input terminals of the memristor array.

[0053] For example, the memristor array includes M source lines, M word lines, N bit lines, and a plurality of memristor units arranged in an array of M rows and N columns.

[0054] For example, the operation of the memristor array is realized through a word line driving circuit, a bit line driving circuit, and a source line driving circuit.

[0055] For example, the word line driving circuit includes a plurality of multiplexers (Multiplexer, referred to as Mux) for switching the word line input voltage; the bit line driving circuit includes a plurality of multiplexers for switching the bit line input voltage; the source line driving circuit also includes a plurality of multiplexers (Mux) for switching the source line input voltage. For example, the source line driving circuit also includes a plurality of ADCs for converting analog signals into digital signals. In addition, a trans-impedance amplifier (Trans-Impedance Amplifier, referred to as TIA) (not shown in the figure) can be further arranged between the Mux and the ADC in the source line driving circuit to complete the current-to-voltage conversion for ADC processing.

[0056] For example, the memristor array includes an operation mode and a calculation mode. When the memristor array is in the operation mode, the memristor unit is in an initialized state, and the values ​​of the parameter elements in the parameter matrix can be written into the memristor array. For example, the source line input voltage, the bit line input voltage, and the word line input voltage of the memristor are switched to the corresponding preset voltage interval through a multiplexer.

[0057] For example, by Figure 2B The control signal WL_sw[1:M] of the multiplexer in the word line driving circuit switches the word line input voltage to the corresponding voltage range. For example, when the memristor is set, the word line input voltage is set to 2V (volts). For example, when the memristor is reset, the word line input voltage is set to 5V. For example, the word line input voltage can be Figure 2B The voltage signal V_WL[1:M] is obtained.

[0058] For example, by Figure 2B The control signal SL_sw[1:M] of the multiplexer in the source line driving circuit switches the source line input voltage to the corresponding voltage range. For example, when the memristor is set, the source line input voltage is set to 0V. For example, when the memristor is reset, the source line input voltage is set to 2V. For example, the source line input voltage can be Figure 2B The voltage signal V_SL[1:M] is obtained.

[0059] For example, by Figure 2B The control signal BL_sw[1:N] of the multiplexer in the bit line driving circuit switches the bit line input voltage to the corresponding voltage range. For example, when the memristor is set, the bit line input voltage is set to 2V, and when the memristor is reset, the bit line input voltage is set to 0V. For example, the bit line input voltage can be Figure 2B The DAC is obtained.

[0060] For example, when the memristor array is in computing mode, the memristors in the memristor array are in a conductive state that can be used for computing, and the bit line input voltage input to the column signal input terminal does not change the conductance value of the memristor. For example, the memristor array can be used to perform multiplication and addition operations to complete the calculation. Figure 2B The control signal WL_sw[1:M] of the multiplexer in the word line driving circuit switches the word line input voltage to the corresponding voltage range, for example, when the start signal is applied, the word line input voltage of the corresponding row is set to 5V, for example, when the start signal is not applied, the word line input voltage of the corresponding row is set to 0V, for example, the GND signal is turned on; Figure 2B The control signal SL_sw[1:M] of the multiplexer in the source line driving circuit switches the source line input voltage to the corresponding voltage interval, for example, setting the source line input voltage to 0V, so that the current signals of the multiple row signal output terminals can flow into the data output circuit, through Figure 2B The control signal BL_sw[1:N] of the multiplexer in the bit line driving circuit switches the bit line input voltage to a corresponding voltage range, for example, setting the bit line input voltage to 0.1V-0.3V, thereby using the memristor array to perform multiplication and addition operations.

[0061] For example, the data output circuit may include a plurality of transimpedance amplifiers (TIAs) and ADCs, which may convert current signals at a plurality of row signal output terminals into voltage signals, and then convert them into digital signals for subsequent processing.

[0062] Figure 2C Schematic diagram of another memristor device. Figure 2C The memristor device shown is similar to Figure 2B The structure of the memristor device shown is basically the same, and also includes a memristor array and its peripheral driving circuit. Figure 2C As shown, the memristor device includes a signal acquisition device, a word line driving circuit, a bit line driving circuit, a source line driving circuit, a memristor array and a data output circuit.

[0063] For example, the memristor array includes M source lines, 2M word lines, and 2N bit lines, and a plurality of memristor units arranged in an array of M rows and N columns. For example, each memristor unit is a 2T2R structure, and the parameter matrix used for transformation processing is mapped to the operation of different multiple memristor units in the memristor array, which will not be repeated here. It should be noted that the memristor array may also include M source lines, M word lines, and 2N bit lines, and a plurality of memristor units arranged in an array of M rows and N columns.

[0064] The description of the signal acquisition device, the control driving circuit and the data output circuit can refer to the previous description and will not be repeated here.

[0065] Figure 2D The process of mapping the weight matrix of the Bayesian neural network to the memristor array is shown. The weight matrix between layers in the Bayesian neural network is realized by using the memristor array. For each weight, N memristors are used to realize the distribution corresponding to the weight, where N is an integer greater than or equal to 2. For the random probability distribution corresponding to the weight, N conductance values ​​are calculated and the N conductance value distributions are mapped to the N memristors. In this way, the weight matrix in the Bayesian neural network is converted into the target conductance value and mapped to the cross sequence of the memristor array.

[0066] like Figure 2D As shown, the left side of the figure is a three-layer Bayesian neural network, which includes three layers of neurons connected one by one. For example, the input layer includes the first layer of neurons, the hidden layer includes the second layer of neurons, and the output layer includes the third layer of neurons. For example, the input layer passes the received input data to the hidden layer, the hidden layer calculates and transforms the input data and sends it to the output layer, and the output layer outputs the output structure of the Bayesian neural network.

[0067] like Figure 2D As shown, the input layer, hidden layer and output layer all include multiple neuron nodes, and the number of neuron nodes in each layer can be set according to different application situations. For example, the number of neurons in the input layer is 2 (including N1 and N2), the number of neurons in the middle hidden layer is 3 (including N3, N4 and N5), and the number of neurons in the output layer is 1 (including N6).

[0068] like Figure 2D As shown in Figure 2, two adjacent neuron layers in a Bayesian neural network are connected by a weight matrix. For example, the weight matrix is ​​composed of Figure 2D The memristor array on the right is implemented. For example, the weight parameter can be directly programmed as the conductance of the memristor array. For example, the weight parameter can also be mapped to the conductance of the memristor array according to a certain rule. For example, the difference in the conductance of two memristors can also be used to represent a weight parameter. Although the present disclosure describes the technical solution of the present disclosure in a manner of directly programming the weight parameter as the conductance of the memristor array or mapping the weight parameter to the conductance of the memristor array according to a certain rule, it is only exemplary and not a limitation of the present disclosure.

[0069] Figure 2D The structure of the memristor array on the right side of the diagram is as follows. Figure 2A As shown, the memristor array may include a plurality of memristors arranged in an array. Figure 2D In the example shown, the weights connecting input N1 and output N3 are composed of three memristors (G 11 , G 12 , G 13 ), and other weights in the weight matrix can be implemented in the same way. More specifically, source line SL1 corresponds to neuron N3, source line SL2 corresponds to neuron N4, source line SL5 corresponds to neuron N5, bit lines BL1, BL2, and BL3 correspond to neuron N1, and a weight between the input layer and the hidden layer (the weight between neuron N1 and neuron N3) is converted into three target conductance values ​​according to the distribution and mapped to the cross sequence of the memristor array. Here, the target conductance values ​​are G 11 , G 12 and G 13 , framed by a dotted line in the memristor array.

[0070] Back to Figure 1B , step S113: input the state and latent input variables corresponding to the dynamic system at time t as input signals to the weight-mapped memristor array, process the state and latent input variables at time t through the memristor array according to the Bayesian neural network, and obtain an output signal corresponding to the processing result from the memristor array, and the output signal is used to obtain the prediction result of the dynamic system at time t+1.

[0071] For example, in some embodiments of the present disclosure, the dynamic environment model is expressed as s t+1 =f(s t ,a t ; W,ε),s t is the state of the dynamic system at time t, a t is the action of the object strategy at time t, W is the weight matrix of the Bayesian neural network, ε is the additive noise corresponding to the memristor array, s t+1 is the prediction result of the dynamic system at time t+1; the action a of the object strategy at time t t =π(s t ; Wπ), π represents the function of the object strategy, Wπ represents the strategy parameter, the weight matrix W of the Bayesian neural network satisfies the distribution W~q(W), and the additive noise ε is the additive Gaussian noise ε~N(0,σ 2 ).

[0072] For a memristor array, the input signal is a voltage signal, the output signal is a current signal, and the output signal is read and converted to digital for subsequent processing. For example, the input sequence is applied to the BL (Bit-line) in the form of a voltage pulse, and then the output current flowing from the SL (Source-line) is collected for further calculation and processing. For example, for Figure 2B Or the memristor device shown in 2C can convert the input sequence into an analog voltage signal through a DAC, and the analog voltage signal is applied to BL through a multiplexer. Correspondingly, the output current is obtained from SL, which can be converted into a voltage signal through a transimpedance amplifier, and converted into a digital signal through an ADC, and the digital signal can be used for subsequent processing. When N memristors read the current and N is relatively large, the total output current presents a certain distribution, such as a distribution similar to a Gaussian distribution or a Laplace distribution. The total output current of all voltage pulses is the result of multiplying the input vector by the weight matrix. In the memristor cross array, such a parallel read operation is equivalent to implementing the two operations of sampling and vector-matrix multiplication.

[0073] Below through Figure 3 It explains how to make multiple predictions at multiple moments based on a dynamic environment model and an object strategy, and obtain a data sample set including the optimization cost of the object strategy corresponding to multiple moments.

[0074] For example, the plurality of time points include time point 1 to time point T arranged in order from earliest to latest.

[0075] Figure 3 Shows Figure 1A Schematic flowchart of an example of step S102 in .

[0076] like Figure 3 As shown, step S102 may include the following steps S301 to S303.

[0077] Step S301: For any time t-1 between time 1 and time T, the object strategy obtains the execution action a t-1 , by a t-1 =π(s t-1 ; Wπ) obtain the action a of the target strategy at time t-1 t-1 .

[0078] For example, action a t-1 is the optimal action selected by the target strategy at the state at time t-1.

[0079] Step S302: According to the dynamic environment model s t =f(s t-1 ,a t-1 ; W, ε) calculates the state s at the next time t after time t-1 t And obtain the state s corresponding to time t t The cost of t , thus obtaining the cost sequence {c1,c2,…,c t }, based on the cost sequence, the optimized cost J at time t is obtained t-1 , where 1≤t≤T.

[0080] For example, in some embodiments of the present disclosure, an example of step S302 may include: sampling the latent input variable z from the p(z) distribution to obtain a sample; comparing the sample and the state s at time t-1 t-1 Input to the weight-mapped memristor array to get the predicted state s t ; For the predicted state s t , get the cost c t =c(s t ).

[0081] For example, first sample the hidden input variable z from the p(z) distribution, and then convert the state s at time t-1 into t-1 The samples of the latent input variables are applied to BL as read voltage pulses of the memristor array, and then the output current flowing out of SL is collected for further calculation and processing to obtain the cost c corresponding to time t t . If the above operation is performed on the state at any time from time 1 to time t, the cost sequence {c1, c2, …, c t }.

[0082] Step S303: Obtain the data sample set {[a0,J0],…,[a T-1 ,J T-1 ]}.

[0083] For example, the cost c at time t t The expected value of t ], then the optimization cost at time t can be obtained by To obtain.

[0084] For example, in some embodiments of the present disclosure, the cost also includes cost changes caused by accidental uncertainty and cost changes caused by epistemic uncertainty, where the accidental uncertainty is caused by hidden input variables and the epistemic uncertainty is caused by the intrinsic noise of the memristor array.

[0085] For example, if we further consider the cost changes caused by accidental uncertainty and epistemic uncertainty, the optimization cost at time t can be expressed as To obtain, σ(η,θ) is a function of aleatoric uncertainty and epistemic uncertainty, η represents aleatoric uncertainty, and θ represents epistemic uncertainty.

[0086] For any moment between time 1 and time T, the data sample corresponding to that moment can be obtained. The data sample set from time 1 to time T is {[a0,J0],…,[a T-1 ,J T-1 ]}.

[0087] For example, the exemplary process of the above-mentioned strategy optimization method of the dynamic environment model based on the memristor array is as follows:

[0088] Input: Dynamic environment model and initial object policy based on memristor array

[0089] n=1 to N, loop

[0090] Initialization state s0

[0091] t=1 to T, loop

[0092] Get the execution action a from the object strategy π t-1

[0093] Using dynamic environment models based on memristor arrays t =f(s t-1 ,a t-1 ; W,ε) predicts the state s at time t t , get the data sample [s t ]

[0094] Calculate the cost and optimization cost corresponding to the object strategy at that moment

[0095] c t =c(s t )

[0096]

[0097] Record [a t-1 ,J t-1 ]

[0098] Get the data sample set {[a0,J0],…,[a T-1 ,J T-1 ]}, and use the policy gradient optimization algorithm to search for strategies on the data sample set

[0099] End the loop of n

[0100] Output: Optimized policy π

[0101] Figure 4 A schematic diagram showing an example of a strategy optimization method provided by at least one embodiment of the present disclosure.

[0102] like Figure 4 As shown in FIG. 1 , in an exemplary application, a ship is driven and thereby fights against the waves to get as close as possible to a target location on the coastline, thereby requiring training of a control model for driving the ship. A ship at position (x, y) can choose an action (a x ,a y ), which indicates the direction and magnitude of the drive. However, due to the dynamic environment of the sea surface with waves, the subsequent position of the ship exhibits drift and disturbance. Moreover, the closer the position is to the coast, the greater the disturbance. The ship is only given a limited batch data set of spatial position transformations, and cannot optimize the action strategy by directly interacting with the ocean environment to ensure safety. At this time, it is necessary to rely on empirical data to learn an ocean environment model (dynamic environment model) that can predict the next state. Epistemic uncertainty and aleatory uncertainty will arise from the missing information of unvisited locations and the randomness of the ocean environment, respectively.

[0103] In this embodiment, for the control model, the sea surface is a dynamic environment, and the object strategy refers to the method used in the process of solving the problem from the current position to the target position of the ship. First, the dynamic environment model and the initial object strategy for the dynamic environment are obtained. The initial state of the ship is the current position, and the execution action at the current moment is obtained by the object strategy. The state at the next moment (the position of the ship) is predicted by the dynamic environment model, and the cost and optimization cost corresponding to the object strategy are calculated, and the data samples composed of the action and the optimization cost are recorded. Assuming that the current moment is moment 1, for moment 1 to the subsequent moment T, a data sample set is obtained, and a policy gradient optimization algorithm is used on the data sample set to search for a strategy, thereby obtaining an optimized object strategy.

[0104] Figure 5 A schematic block diagram of a strategy optimization device 500 using a dynamic environment model based on a memristor array provided by at least one embodiment of the present disclosure is shown. The strategy optimization device can be used to perform Figure 1A The data processing method shown.

[0105] like Figure 5 As shown, the policy optimization device 500 includes an acquisition unit 501 , a calculation unit 502 and a policy search unit 503 .

[0106] The acquisition unit 501 is configured to acquire a dynamic environment model based on a memristor array.

[0107] The calculation unit 502 is configured to perform multiple predictions at multiple time points according to the dynamic environment model and the object strategy, and obtain a data sample set including the optimization costs of the object strategy corresponding to the multiple time points.

[0108] The policy search unit 503 is configured to perform policy search based on the data sample set using a policy gradient optimization algorithm to optimize the object policy.

[0109] For example, the policy optimization device 500 may be implemented using hardware, software, firmware, or any feasible combination thereof, and the present disclosure does not limit this.

[0110] The technical effect of the above strategy optimization device is Figure 1A The technical effects of the strategy optimization method shown are the same and will not be repeated here.

[0111] There are a few points to note:

[0112] (1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure, and other structures may refer to the general design.

[0113] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0114] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.< / m> < / n> < / m>

Claims

1. A strategy optimization method using a dynamic environment model based on a memristor array, include: Acquiring the dynamic environment model based on the memristor array; Perform multiple predictions at multiple moments according to the dynamic environment model and the object strategy to obtain a data sample set including the optimization cost of the object strategy corresponding to the multiple moments; Based on the data sample set, using a policy gradient optimization algorithm to perform policy search to optimize the object policy; Wherein, obtaining the dynamic environment model includes: Obtaining a Bayesian neural network, wherein the Bayesian neural network has a trained weight matrix, According to the weight matrix of the Bayesian neural network, a corresponding plurality of target conductance values ​​are obtained, and the plurality of target conductance values ​​are mapped to the memristor array. Inputting the state and hidden input variables corresponding to the time t of the dynamic system as input signals into the weight-mapped memristor array, processing the state and hidden input variables at the time t according to the Bayesian neural network through the memristor array, and obtaining an output signal corresponding to the processing result from the memristor array, wherein the output signal is used to obtain a prediction result of the dynamic system at the time t+1; The dynamic environment model is expressed as s t+1 = f(s t , a t ; W, ε), Among them, s t is the state of the dynamic system at time t, a t is the action of the object strategy at time t, W is the weight matrix of the Bayesian neural network, ε is the additive noise corresponding to the memristor array, s t+1 is the prediction result of the dynamic system at the time t+1, Among them, the action a of the object strategy at time t t =π(s t ; Wπ), π represents the function of the object strategy, Wπ represents the strategy parameter, the weight matrix W of the Bayesian neural network satisfies the distribution W~q(W), and the additive noise ε is the additive Gaussian noise ε∼N(0, σ 2 ).

2. The strategy optimization method according to claim 1, in, The multiple time points include time 1 to time T arranged in order from early to late, Performing multiple predictions at the multiple moments according to the dynamic environment model and the object strategy to obtain the data sample set including the optimization cost of the object strategy corresponding to the multiple moments, includes: For any time t-1 between the time 1 and the time T, the object strategy obtains the execution action a t-1 , By a t-1 =π(s t-1 Wπ) obtain the action a of the object strategy at the time t-1 t-1 ; According to the dynamic environment model s t = f(s t-1 ,a t-1 ; W, ε) calculates the state s at the next moment t after the moment t-1 t And obtain the state s corresponding to the time t t The cost of t , thus obtaining the cost sequence {c 1 , c 2 , …, c t }, Based on the cost sequence, the optimized cost J at the time t is obtained t-1 , where 1≤t≤T; Get the data sample set from time 1 to time T {[a 0 ,J 0 ],…,[a T-1 ,J T-1 ]}.

3. The strategy optimization method according to claim 2, in, The cost c at the time t t The expected value of t ], then the optimization cost at time t can be obtained by To obtain.

4. The strategy optimization method according to claim 2, in, The cost also includes the cost changes caused by accidental uncertainty and the cost changes caused by cognitive uncertainty. The aleatory uncertainty is caused by the implicit input variable, and the epistemic uncertainty is caused by the intrinsic noise of the memristor array.

5. The strategy optimization method according to claim 4, in, The optimization cost at time t is given by To obtain, among which, is a function of the aleatoric uncertainty and the epistemic uncertainty, η represents the aleatoric uncertainty, and θ represents the epistemic uncertainty.

6. The strategy optimization method according to claim 2, in, According to the dynamic environment model s t = f(s t-1 ,a t-1 ;W, ε) calculate the state s at the time t t And obtain the state s corresponding to the time t t The cost c t ,include: Sampling the latent input variable z from the p(z) distribution to obtain a sample; The sample and the state s at time t-1 t-1 Input to the weight-mapped memristor array to obtain the predicted state s t ; For the predicted state s t , obtain the cost c t =c(s t ).

7. The strategy optimization method according to claim 1, in, The policy gradient optimization algorithm includes the REINFORCE algorithm, the PRO algorithm or the TPRO algorithm.

8. A strategy optimization device using a dynamic environment model based on a memristor array, include: An acquisition unit configured to acquire the dynamic environment model based on the memristor array; A computing unit configured to perform multiple predictions at multiple moments according to the dynamic environment model and the object strategy, and obtain a data sample set including the optimization cost of the object strategy corresponding to the multiple moments; A policy search unit, configured to perform policy search based on the data sample set using a policy gradient optimization algorithm to optimize the object policy; Wherein, when acquiring the dynamic environment model, the acquiring unit is configured to: Obtaining a Bayesian neural network, wherein the Bayesian neural network has a trained weight matrix, According to the weight matrix of the Bayesian neural network, a corresponding plurality of target conductance values ​​are obtained, and the plurality of target conductance values ​​are mapped to the memristor array. Inputting the state and hidden input variables corresponding to the time t of the dynamic system as input signals into the weight-mapped memristor array, processing the state and hidden input variables at the time t according to the Bayesian neural network through the memristor array, and obtaining an output signal corresponding to the processing result from the memristor array, wherein the output signal is used to obtain a prediction result of the dynamic system at the time t+1; The dynamic environment model is expressed as s t+1 = f(s t , a t ; W, ε), Among them, s t is the state of the dynamic system at time t, a t is the action of the object strategy at time t, W is the weight matrix of the Bayesian neural network, ε is the additive noise corresponding to the memristor array, s t+1 is the prediction result of the dynamic system at the time t+1, Among them, the action a of the object strategy at time t t =π(s t ; Wπ), π represents the function of the object strategy, Wπ represents the strategy parameter, the weight matrix W of the Bayesian neural network satisfies the distribution W~q(W), and the additive noise ε is the additive Gaussian noise ε∼N(0, σ 2 ).