Text sequence prediction method and system, storage medium and electronic equipment

Abstract

The invention provides a text sequence prediction method and system, a storage medium and electronic equipment. The method comprises the following steps: constructing an energy function of a text sequence prediction neural network; constructing a continuous time dynamic equation of a neuron state based on a network function; inputting the text lexical element vector of the current time step into a text sequence prediction neural network, and evolving the text sequence prediction neural network to a first equilibrium state based on a continuous time dynamic equation; adjusting an output layer neuron state of the text sequence prediction neural network into a text lexical element vector of a next time step based on a continuous time dynamic equation, so that the text sequence prediction neural network is evolved to a second equilibrium state; and updating the weight of the text sequence prediction neural network based on the neuron state difference in the first equilibrium state and the second equilibrium state. According to the text sequence prediction method and system, the storage medium and the electronic equipment, rapid prediction of the text sequence is realized based on the neural network evolved by the energy function and the continuous time dynamic equation.

Description

Technical Field

[0001] This invention belongs to the technical field of deep learning, and in particular relates to a text sequence prediction method, system, storage medium, and electronic device. Background Technology

[0002] Neural networks, especially recurrent neural networks (RNNs), have achieved significant success in processing temporal data tasks such as speech recognition, natural language processing, and video analysis. Currently, the mainstream algorithm for training RNNs is the backpropagation algorithm based on time propagation. However, this algorithm has the following drawbacks:

[0003] (1) The algorithm requires the network to be deployed over the entire time series, resulting in huge memory consumption and computational cost, making it difficult to process long sequence data;

[0004] (2) The algorithm relies on the backpropagation of global error signals, which is inconsistent with the local learning mechanism of biological nervous systems, and its biological rationality is highly questionable.

[0005] (3) In deep networks or long sequences, this algorithm is prone to gradient vanishing or gradient explosion problems, which affect the stability and effectiveness of training.

[0006] In existing technologies, when methods that truncate backpropagation paths are used, the model’s ability to capture long-term dependencies is usually compromised. When biologically inspired local learning rules are used, although they have biological rationale, they usually lack a solid theoretical foundation and optimization guarantees, and their performance on complex tasks is often inferior to traditional methods.

[0007] Therefore, existing neural network algorithms cannot meet the needs of practical applications when processing long text sequences. Summary of the Invention

[0008] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a text sequence prediction method, system, storage medium and electronic device, which realizes fast prediction of text sequences based on a neural network of energy function and continuous-time dynamic equation evolution.

[0009] In a first aspect, the present invention provides a text sequence prediction method, the method comprising the following steps: constructing an energy function of a text sequence prediction neural network; constructing a continuous-time dynamic equation of the neuron states in the text sequence prediction neural network based on the network function; inputting the text word vector at the current time step into the text sequence prediction neural network, and evolving the text sequence prediction neural network to a first equilibrium state based on the continuous-time dynamic equation; adjusting the output layer neuron states of the text sequence prediction neural network to the text word vector at the next time step based on the continuous-time dynamic equation, so that the text sequence prediction neural network evolves to a second equilibrium state; updating the weights of the text sequence prediction neural network based on the difference in neuron states between the first equilibrium state and the second equilibrium state, until the text sequence prediction neural network converges, so as to obtain text word prediction results based on the trained text sequence prediction neural network.

[0010] In one implementation of the first aspect, the energy function Where s i and s j Let x represent the state vectors of neurons i and j, respectively. i Let y represent the input vector of neuron i, y represent the target value vector corresponding to the output layer neuron, and w represent the input vector of neuron i. ij ρ represents the weight connecting neuron i and neuron j, ρ(·) represents the neuron activation function, β represents the clamping coefficient, and k represents the index of the output layer neuron.

[0011] In one implementation of the first aspect, the continuous-time dynamic equation Where τ represents the time constant, s i Let E represent the state vector of neuron i, and let E represent the energy function.

[0012] In one implementation of the first aspect, inputting the text word vector at the current time step into the text sequence prediction neural network, and evolving the text sequence prediction neural network to a first equilibrium state based on the continuous-time dynamic equation includes the following steps:

[0013] Set the clamping coefficient of the energy function to 0;

[0014] The text word vector at the current time step is input into the text sequence prediction neural network, and the neuron state of the text prediction sequence neural network is evolved based on the continuous-time dynamic equation until the energy function reaches a local minimum.

[0015] In one implementation of the first aspect, adjusting the output layer neuron state of the text sequence prediction neural network to the text word vector of the next time step based on the continuous-time dynamic equation, so that the text sequence prediction neural network evolves to a second equilibrium state, includes the following steps:

[0016] Set the clamping coefficient of the energy function to a value greater than 0;

[0017] The neuron states of the text prediction sequence neural network are evolved based on the continuous-time dynamic equation until the output layer neuron states of the text sequence prediction neural network are adjusted to the text word vectors of the next time step, so that the text sequence prediction neural network evolves to the second equilibrium state.

[0018] In one implementation of the first aspect, updating the weights of the text sequence prediction neural network based on the difference in neuron states under the first equilibrium state and the second equilibrium state includes the following steps:

[0019] Get weight update amount Where Δw ij η represents the weight update amount, η represents the learning rate, and s represents the learning rate. free s represents the neuron state in the first equilibrium state. clamp Let ρ(·) represent the neuron state in the second equilibrium state, and let ρ(·) represent the neuron activation function.

[0020] The weights of the text sequence prediction neural network are updated based on the weight update amount.

[0021] In one implementation of the first aspect, the neurons of a text sequence prediction neural network are represented by analog circuit units, wherein the capacitor voltage in the analog circuit units represents the neuron state.

[0022] The conductance values ​​based on memristors represent the weights of the text sequence prediction neural network;

[0023] The conductance of the memristor is adjusted by monitoring the voltage difference between the analog circuit units across the memristor.

[0024] Secondly, the present invention provides a text sequence prediction system for text standardization expression proofreading, the system comprising a first construction module, a second construction module, a free training module, a clamped training module, and an update module;

[0025] The first building module is used to construct the energy function of a text sequence prediction neural network;

[0026] The second building module is used to construct the continuous-time dynamic equation of the neuron state in the text sequence prediction neural network based on the network function;

[0027] The free training module is used to input the text word vector of the current time step into the text sequence prediction neural network, and evolve the text sequence prediction neural network to the first equilibrium state based on the continuous-time dynamic equation;

[0028] The clamping training module is used to adjust the output layer neuron state of the text sequence prediction neural network to the text word vector of the next time step based on the continuous-time dynamic equation, so that the text sequence prediction neural network evolves to the second equilibrium state.

[0029] The update module is used to update the weights of the text sequence prediction neural network based on the difference in neuron states under the first equilibrium state and the second equilibrium state, until the text sequence prediction neural network converges, so as to obtain text word prediction results based on the trained text sequence prediction neural network.

[0030] Thirdly, the present invention provides an electronic device, the electronic device comprising: a processor and a memory;

[0031] The memory is used to store computer programs;

[0032] The processor is used to execute the computer program stored in the memory to cause the electronic device to perform the text sequence prediction method described above.

[0033] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-described text sequence prediction method.

[0034] As described above, the text sequence prediction method, system, storage medium, and electronic device of the present invention have the following beneficial effects:

[0035] (1) A neural network based on the evolution of energy function and continuous-time dynamic equation realizes fast prediction of text sequence;

[0036] (2) In the training process of the text sequence prediction neural network, learning is achieved through a two-stage energy balance process, which avoids the network expansion and backpropagation errors in the time dimension, significantly reducing memory consumption and computational costs, especially when processing long sequence data; the weight update rule is completely local and depends only on the activity state of neurons before and after the synapse, which is highly consistent with the learning mechanism in biological neural systems; it provides a theoretical basis for building an intelligent system that is closer to the biological brain; the training method has been proven to be gradient descent on a well-defined objective function, which ensures the convergence and stability of the learning process and solves the problem that many bio-inspired learning algorithms lack theoretical guarantees;

[0037] (3) The continuous dynamic system and local learning rules of the present invention are very suitable for implementation on analog circuits or neuromorphic hardware, which can give full play to the advantages of these hardware in terms of energy efficiency and parallel computing, and provide a feasible algorithm solution for developing low-power, high-performance edge intelligent devices.

[0038] (4) It solves the problems of high computational resource consumption and difficulty in gradient propagation in the training of large language models based on Transformer or RNN in the existing technology. Attached Figure Description

[0039] Figure 1 The flowchart shown is an embodiment of the text sequence prediction method of the present invention;

[0040] Figure 2 The diagram shows an architecture schematic of the text sequence prediction method of the present invention in one embodiment;

[0041] Figure 3 The diagram shown is a structural schematic of the text sequence prediction system of the present invention in one embodiment.

[0042] Figure 4 The diagram shown is a structural schematic of an embodiment of the electronic device of the present invention. Detailed Implementation

[0043] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0044] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0045] The following embodiments of the present invention provide a text sequence prediction method, which can be applied to, for example... Figure 1The electronic devices shown are as follows. The electronic devices described in this invention may include mobile phones 11 with wireless charging capabilities, tablet computers 12, laptop computers 13, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. This invention does not impose any restrictions on the specific type of electronic device.

[0046] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0047] like Figure 1 As shown, in one embodiment, the text sequence prediction method of the present invention includes steps S1-S5.

[0048] Step S1: Construct the energy function of the text sequence prediction neural network.

[0049] Specifically, the text sequence prediction neural network is considered as a physical system, and its overall state is described by an energy function. In one embodiment, the energy function... Where s i and s j Let x represent the state vectors of neurons i and j, respectively. i Let y represent the input vector of neuron i, y represent the target value vector corresponding to the output layer neuron, and w represent the input vector of neuron i. ij ρ represents the weight connecting neurons i and j, ρ(·) represents the neuron activation function, β represents the clamping coefficient (which varies depending on the application), and k represents the index of the output layer neuron.

[0050] Step S2: Construct the continuous-time dynamic equation of the neuron state in the text sequence prediction neural network based on the network function.

[0051] Specifically, the state s of the neuron is modeled as a continuous variable that evolves over time, with the evolution rule following the negative gradient direction of the energy function to minimize the system energy. In one embodiment, the continuous-time dynamic equation... Where τ represents the time constant, used to control the rate of state evolution. i Let represent the state vector of neuron i, and E represent the energy function. The continuous-time dynamic equation describes how the state of each neuron adjusts according to its own state, connection weights, and external input.

[0052] Step S3: Input the text word vector of the current time step into the text sequence prediction neural network, and evolve the text sequence prediction neural network to the first equilibrium state based on the continuous time dynamic equation.

[0053] Specifically, this step is the free training phase. In this phase, the text sequence prediction neural network only receives the text word vector of the current time step, without considering the target value vector (i.e., the text word vector of the next time step). First, the clamping coefficient β is set to 0. The text sequence prediction neural network starts from the initial state, inputting the text word vector of the current time step. Based on the continuous-time dynamic equation, the neuron states of the text sequence prediction neural network evolve. After a period of time, the system will reach a stable equilibrium state, that is, the energy function reaches a local minimum. The rate of change of the neuron state approaches zero. The neuron state at this time is recorded and called the first equilibrium state s. free .

[0054] Step S4: Based on the continuous-time dynamic equation, adjust the output layer neuron state of the text sequence prediction neural network to the text word vector of the next time step, so that the text sequence prediction neural network evolves to the second equilibrium state.

[0055] Specifically, this step is the clamping training phase. In this phase, the clamping coefficient of the energy function is set to a value greater than 0, thereby introducing the target value vector into the energy function. This is equivalent to applying a small push to the output layer, guiding the state of the output layer neurons towards the target value vector. The text sequence prediction neural network starts from the state s at the end of the free training phase. free Initially, the neuron states of the text prediction sequence neural network are evolved based on the continuous-time dynamic equation until the output layer neuron states of the text sequence prediction neural network are adjusted to the text word vectors of the next time step, so that the text sequence prediction neural network evolves to the second equilibrium state. The neuron state at this time is recorded and called the second equilibrium state s. clamp .

[0056] Step S5: Update the weights of the text sequence prediction neural network based on the differences in neuron states under the first equilibrium state and the second equilibrium state until the text sequence prediction neural network converges, so as to obtain the text word prediction result based on the trained text sequence prediction neural network.

[0057] Specifically, the weights are updated based on the difference in neuronal activity between two equilibrium states: the free training phase and the clamped training phase. This update rule is local, meaning that the adjustment of each weight depends only on the states of the two neurons it is connected to. In one embodiment, the weight update amount is first obtained. Where Δw ij η represents the weight update amount, η represents the learning rate, and s represents the learning rate. free s represents the neuron state in the first equilibrium state. clamp Let ρ(·) represent the neuron state in the second equilibrium state, and let ρ(·) represent the neuron activation function. The weights of the text sequence prediction neural network are then updated based on the weight update amount. The updated weights are equal to the sum of the current weights and the weight update amount. The physical meaning of the above weight update rule is: if the activation product of two neurons in the clamped training phase is greater than the activation product in the free training phase, the connection between them is strengthened; otherwise, it is weakened. This process is equivalent to performing gradient descent on the objective function. Repeat steps S3-S5 until the text sequence prediction neural network converges. Subsequently, based on the trained text sequence prediction neural network, the next text word vector can be predicted for the current input text word vector, thereby achieving accurate text sequence prediction.

[0058] It should be noted that for a text sequence, the input text word vector x(t) changes over time. At each time step of processing the text sequence, the network state s(t) evolves from the ending state of the previous time step, thus naturally incorporating historical information into the current state and effectively capturing time dependencies. Furthermore, a leakage factor or a time decay factor can be introduced to control the degree of influence of historical information.

[0059] like Figure 2 As shown, the text sequence prediction method for text specification expression correction of the present invention can be implemented based on a neuromorphic computing device. Analog circuit units represent neurons in the text sequence prediction neural network, where the capacitor voltage in the analog circuit unit represents the neuron state. The conductance value of the memristor represents the weights of the text sequence prediction neural network. The energy function and continuous-time dynamic equation are naturally implemented in the circuit using Kirchhoff's laws, with the current flow corresponding to the energy gradient descent. The switching between the free training phase and the clamped training phase is achieved through an external control signal. In the clamped training phase, the control signal applies an additional bias voltage to the circuit corresponding to the output layer neuron, thereby clamping the output state. The weight update module monitors the voltage of the analog circuit unit across each memristor in both phases and adjusts the conductance value of the memristor based on the voltage difference. The entire learning process is performed entirely in the analog domain in a distributed and asynchronous manner, without the need for centralized clock signals and digital computing units, thus achieving extremely high energy efficiency.

[0060] The scope of protection of the text sequence prediction method described in this embodiment is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principle of this invention is included within the scope of protection of this invention.

[0061] This invention also provides a text sequence prediction system, which can implement the text sequence prediction method described in this invention. However, the implementation device of the text sequence prediction system described in this invention includes, but is not limited to, the structure of the text sequence prediction system listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this invention are included within the protection scope of this invention.

[0062] like Figure 3 As shown, in one embodiment, the text sequence prediction system of the present invention includes a first construction module 31, a second construction module 32, a free training module 33, a clamped training module 34, and an update module 35.

[0063] The first building module 31 is used to build the energy function of the text sequence prediction neural network.

[0064] The second construction module 32 is connected to the first construction module 31 and is used to construct the continuous-time dynamic equation of the neuron state in the text sequence prediction neural network based on the network function.

[0065] The free training module 33 is connected to the second construction module 32 and is used to input the text word vector of the current time step into the text sequence prediction neural network and evolve the text sequence prediction neural network to the first equilibrium state based on the continuous time dynamic equation.

[0066] The clamping training module 34 is connected to the free training module 33 and is used to adjust the output layer neuron state of the text sequence prediction neural network to the text word vector of the next time step based on the continuous-time dynamic equation, so that the text sequence prediction neural network evolves to the second equilibrium state.

[0067] The update module 35 is connected to the clamping training module 34 and is used to update the weights of the text sequence prediction neural network based on the difference in neuron states under the first equilibrium state and the second equilibrium state until the text sequence prediction neural network converges, so as to obtain the text word prediction result based on the trained text sequence prediction neural network.

[0068] The structure and principle of the first construction module 31, the second construction module 32, the free training module 33, the clamping training module 34, and the update module 35 correspond one-to-one with the steps in the above text sequence prediction method, so they will not be described again here.

[0069] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0070] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs. For example, the functional modules / units in the various embodiments of the present invention may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0071] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0072] This invention also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. This available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0073] This invention also provides an electronic device. The electronic device includes a processor and a memory.

[0074] The memory is used to store computer programs.

[0075] The memory includes various media capable of storing program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card, or optical disk.

[0076] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device performs the above-described text sequence prediction method.

[0077] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0078] like Figure 4As shown, the electronic device of the present invention is embodied in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors or processing units 41, a memory 42, and a bus 43 connecting different system components (including the memory 42 and the processing unit 41).

[0079] Bus 43 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0080] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0081] Memory 42 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 421 and / or cache memory 422. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 423 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 43 via one or more data media interfaces. Memory 42 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0082] A program / utility 424 having a set (at least one) of program modules 4241 may be stored, for example, in memory 42. Such program modules 4241 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 4241 typically perform the functions and / or methods described in the embodiments of the present invention.

[0083] The electronic device can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 44. Furthermore, the electronic device can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 45. Figure 4 As shown, network adapter 45 communicates with other modules of the electronic device via bus 43. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0084] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A text sequence prediction method, characterized in that, The method includes the following steps: Constructing an energy function for a text sequence prediction neural network; Based on the network function, construct the continuous-time dynamic equation of the neuron state in the text sequence prediction neural network; The text word vector at the current time step is input into the text sequence prediction neural network, and the text sequence prediction neural network is evolved to the first equilibrium state based on the continuous-time dynamic equation; Based on the continuous-time dynamic equation, the output layer neuron state of the text sequence prediction neural network is adjusted to the text word vector of the next time step, so that the text sequence prediction neural network evolves to the second equilibrium state; The weights of the text sequence prediction neural network are updated based on the differences in neuron states under the first equilibrium state and the second equilibrium state until the text sequence prediction neural network converges, so as to obtain text word prediction results based on the trained text sequence prediction neural network.

2. The text sequence prediction method according to claim 1, characterized in that: The energy function Where s i and s j Let x represent the state vectors of neurons i and j, respectively. i Let y represent the input vector of neuron i, y represent the target value vector corresponding to the output layer neuron, and w represent the input vector of neuron i. ij ρ represents the weight connecting neuron i and neuron j, ρ(·) represents the neuron activation function, β represents the clamping coefficient, and k represents the index of the output layer neuron.

3. The text sequence prediction method according to claim 1, characterized in that: The continuous-time dynamic equation Where τ represents the time constant, s i Let E represent the state vector of neuron i, and let E represent the energy function.

4. The text sequence prediction method according to claim 2, characterized in that: Inputting the text word vector at the current time step into the text sequence prediction neural network, and evolving the text sequence prediction neural network to the first equilibrium state based on the continuous-time dynamic equation includes the following steps: Set the clamping coefficient of the energy function to 0; The text word vector at the current time step is input into the text sequence prediction neural network, and the neuron state of the text prediction sequence neural network is evolved based on the continuous-time dynamic equation until the energy function reaches a local minimum.

5. The text sequence prediction method according to claim 2, characterized in that: Adjusting the output layer neuron state of the text sequence prediction neural network to the text word vector of the next time step based on the continuous-time dynamic equation, so that the text sequence prediction neural network evolves to the second equilibrium state, includes the following steps: Set the clamping coefficient of the energy function to a value greater than 0; The neuron states of the text prediction sequence neural network are evolved based on the continuous-time dynamic equation until the output layer neuron states of the text sequence prediction neural network are adjusted to the text word vectors of the next time step, so that the text sequence prediction neural network evolves to the second equilibrium state.

6. The text sequence prediction method according to claim 1, characterized in that: Updating the weights of the text sequence prediction neural network based on the differences in neuron states between the first and second equilibrium states includes the following steps: Get weight update amount Where Δw ij η represents the weight update amount, η represents the learning rate, and s represents the learning rate. free s represents the neuron state in the first equilibrium state. clamp Let ρ(·) represent the neuron state in the second equilibrium state, and let ρ(·) represent the neuron activation function. The weights of the text sequence prediction neural network are updated based on the weight update amount.

7. The text sequence prediction method according to claim 1, characterized in that: The neurons of a text sequence prediction neural network are represented by analog circuit units, wherein the capacitor voltage in the analog circuit unit represents the neuron state. The conductance values ​​based on memristors represent the weights of the text sequence prediction neural network; The conductance of the memristor is adjusted by monitoring the voltage difference between the analog circuit units across the memristor.

8. A text sequence prediction system, characterized in that, The system includes a first construction module, a second construction module, a free training module, a clamping training module, and an update module; The first building module is used to construct the energy function of a text sequence prediction neural network; The second building module is used to construct the continuous-time dynamic equation of the neuron state in the text sequence prediction neural network based on the network function; The free training module is used to input the text word vector of the current time step into the text sequence prediction neural network, and evolve the text sequence prediction neural network to the first equilibrium state based on the continuous-time dynamic equation; The clamping training module is used to adjust the output layer neuron state of the text sequence prediction neural network to the text word vector of the next time step based on the continuous-time dynamic equation, so that the text sequence prediction neural network evolves to the second equilibrium state. The update module is used to update the weights of the text sequence prediction neural network based on the difference in neuron states under the first equilibrium state and the second equilibrium state, until the text sequence prediction neural network converges, so as to obtain text word prediction results based on the trained text sequence prediction neural network.

9. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory to cause the electronic device to perform the text sequence prediction method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by an electronic device, the program implements the text sequence prediction method according to any one of claims 1 to 7.

Images