GRU network model based on ordinary differential equations and feature extraction method and device

By using ordinary differential equations in GRU network to use the derivative of hidden states as continuous parameters, the problem of storage space efficiency in deep learning models is solved, memory efficiency improvement and automatic connection of model parameters is achieved, and memory in the existing technology is solved A big problem of occupancy.

CN113792860BActive Publication Date: 2024-11-19THE SECOND AFFILIATED HOSPITAL ARMY MEDICAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111080690.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-15
Publication Date
2024-11-19
Estimated Expiration
2041-09-15

AI Technical Summary

Technical Problem

In the existing deep learning model, GRU networks need a lot of storage space to save intermediate results during training, resulting in high memory footprint and affecting efficiency.

Method used

The GRU network model based on ordinary differential equations is adopted, and the derivative of the hidden state is used as a continuous parameter to reduce the storage space requirement, and the calculation is performed through the Euler method or the Longge-Kuta method to automatically connect the network layer parameters to reduce the number of parameters.

Benefits of technology

It significantly reduces the storage space requirement, improves memory efficiency, and automatically connects the network layer parameters by parameterizing the hidden layer as a time continuous function, reducing the number of parameters of the supervised learning task, and improving the training efficiency and accuracy of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113792860B_ABST
    Figure CN113792860B_ABST
Patent Text Reader

Abstract

This invention discloses a GRU network model and feature extraction method and apparatus based on ordinary differential equations. The GRU network model includes at least one gated recurrent unit, and each gated recurrent unit includes a candidate hidden layer. The candidate hidden layer calculates its input information based on ordinary differential equations and outputs a sequence of hidden states. This GRU network model does not require a large amount of space to store intermediate results during training, reducing storage space and improving memory efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a GRU network model based on ordinary differential equations and a feature extraction method and device. Background Art

[0002] In recent years, deep learning has developed rapidly, and it has become a sensation due to Google's AlphaGo some time ago. China has also begun to usher in a research boom in this technology. Deep learning is still in the development stage. Researchers have applied deep learning to related image processing and computer vision fields. Deep learning algorithms have achieved outstanding results in many supervised learning problems. Its performance in terms of accuracy, efficiency and intelligence far exceeds that of traditional machine learning algorithms, and some even exceed the human level. At present, the research interests of deep learning researchers have gradually shifted from supervised learning to reinforcement learning, semi-supervised learning and unsupervised learning. Since 2006, deep learning has continued to heat up in academia, and the United States, Canada, and Europe have successively become research centers in this field. In 2013, the European Commission launched a supercomputer project that imitates the human brain. It plans to invest $1.6 billion over 10 years and is jointly participated by more than 200 researchers from 80 institutions around the world. It hopes to make significant progress in understanding how the human brain works and promote the development of more powerful new computers.

[0003] Current multi-layer neural networks, such as the GRU network model, are composed of multiple repeated gated recurrent units in series. Gated Recurrent Units (GRU) are a gating mechanism in the RNN. Similar to other gating mechanisms, they aim to solve the gradient vanishing / exploding problem in the standard RNN while retaining the long-term information of the sequence. Each gated recurrent unit in the GRU network model retains certain information in an orderly manner and is optimized during the training process.

[0004] The essence of neural networks is to fit a complex composite function. The number of composite functions is the number of layers in the neural network. To find the gradient of the parameters, it is easy to think of the chain rule. However, it is necessary to retain the activation values ​​of all layers during forward propagation and use these activation values ​​during back propagation. This takes up a lot of memory and is a big bottleneck for the training process of deep models. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a GRU network model based on ordinary differential equations and a feature extraction method and device, which does not require a large amount of space to store intermediate results during the training process, reduces storage space and improves memory efficiency.

[0006] The present invention is achieved through the following technical solutions:

[0007] A first aspect of the present invention provides a GRU network model based on ordinary differential equations, comprising at least one gated recurrent unit, each of the at least one gated recurrent unit comprising a candidate hidden layer, and the candidate hidden layer calculates input information of the candidate hidden layer based on the ordinary differential equation and converts it into a hidden state sequence output.

[0008] The first aspect of the present invention improves the candidate hidden layer, that is, the candidate hidden layer adopts ordinary differential equations, combines the GRU network model of deep learning with ordinary differential equations, and uses the derivative of the hidden state as a parameter. The parameter is not a series of discrete values, but a continuous space, so it does not need to be passed to each function in the forward propagation for evaluation in turn, so that a large amount of space is not consumed to store intermediate results, which greatly reduces the storage space and improves memory efficiency. The parameter efficiency is high. By solving ordinary differential equations, when the hidden layer dynamics are parameterized as a continuous function of time, the parameters of the nearby "network layer" are automatically connected together, thereby reducing the number of parameters required for supervised learning tasks.

[0009] In a possible design, the candidate hidden layer uses the Euler method or the Runge-Kutta method to calculate the input information of the candidate hidden layer and convert it into a hidden state sequence output.

[0010] A second aspect of the present invention provides a feature extraction method, comprising the following steps:

[0011] Acquire at least one image, each of the at least one image including at least one target to be identified;

[0012] Convolution and pooling are performed on each of the at least one image, and then the convolution and pooling are performed on the ...

[0013] The second aspect of the present invention discloses the application of the GRU network model in the first aspect. After convolution pooling of an image with a target to be identified, the GRU network model is used to realize feature extraction of the target to be identified, thereby reducing the memory usage of the GRU network model during the training process.

[0014] In a possible design, after acquiring at least one image, the method further includes: a step of preprocessing each of the at least one image.

[0015] In a possible design, the step of preprocessing each of the at least one image includes:

[0016] The holes in each image of the at least one image are filled using a bilateral filtering algorithm.

[0017] This solution uses a bilateral filtering algorithm to fill the holes in each image, so that the preprocessed image is clearer, the feature extraction is more accurate, the recognition rate is improved, and the accuracy of feature extraction is further improved.

[0018] In one possible design, the image is an RGB-D image.

[0019] Existing feature extraction is generally based on two-dimensional RGB images. This solution preferably uses RGB-D images, which are depth images. Their changes in light intensity are not affected by changes in color and texture. They can be reliably used for feature extraction and provide rich structural information of 3D scenes.

[0020] In one possible design, a convolutional neural network is used to perform convolution and pooling on each of the at least one image.

[0021] The third aspect of the present invention discloses a feature extraction device, comprising a memory and a controller which are communicatively connected in sequence, wherein a computer program is stored in the memory, and the controller is used to read the computer program and execute a feature extraction method described in the second aspect and any possible one thereof.

[0022] A fourth aspect of the present invention discloses a computer-readable storage medium, on which instructions are stored. When the instructions are run on a computer, a feature extraction method described in the second aspect and any possible one thereof is executed.

[0023] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0024] 1. The candidate hidden layer of the GRU network model of the present invention adopts an ordinary differential equation, and takes the derivative of the hidden state as a parameter. The parameter is changed from an existing discrete value to a continuous value. During training, it does not need to be passed to each function in the forward propagation for evaluation in turn, so that a large amount of space is not consumed to store the intermediate results, which greatly reduces the storage space and improves the memory efficiency.

[0025] 2. The feature extraction method of the present invention realizes feature extraction of the target to be identified based on the above-mentioned ordinary differential-GRU network model, reducing the memory usage of the GRU network model during the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention.

[0027] Figure 1 This is the architecture diagram of the gated recurrent unit.

[0028] Figure 2 Flow chart of the feature extraction method of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be noted that although the description of these embodiments is used to help understand the present invention, it does not constitute a limitation of the present invention. The specific structures and functional details disclosed herein are only used to describe the exemplary embodiments of the present invention. However, the present invention can be embodied in many alternative forms, and it should not be understood that the present invention is limited to the embodiments set forth herein.

[0030] It should be understood that although the terms first, second, etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another unit. For example, a first unit can be referred to as a second unit, and similarly, a second unit can be referred to as a first unit without departing from the scope of the exemplary embodiments of the present invention.

[0031] It should be understood that the term "and / or" that may appear in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" that may appear in this article describes another type of association object relationship, indicating that two relationships may exist. For example, A / and B can represent two situations: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this article generally indicates that the previous and next associated objects are in an "or" relationship.

[0032] It should be understood that certain details are provided in the following description to facilitate a complete understanding of the example embodiments. However, one of ordinary skill in the art should understand that the example embodiments can be implemented without these specific details. For example, the system can be shown in a block diagram to avoid obscuring the example with unnecessary details. In other examples, well-known processes, structures, and techniques may not be shown in unnecessary detail to avoid obscuring the example embodiments.

[0033] The first aspect of the present invention discloses a GRU network model based on ordinary differential equations, including at least one gated recurrent unit. When there are multiple gated recurrent units, the multiple gated recurrent units are connected in series.

[0034] The gated recurrent unit includes a reset gate r t , update gate z t and candidate hidden layers Reset Gate t , update gate z t and candidate hidden layers The connection is prior art, refer to Figure 1 That's it, I won't go into details here, reset gate r t , update gate z t and candidate hidden layers All use the state information h of the previous moment t-1 and the original data as input. In this scheme, the candidate hidden layer of each gated recurrent unit The input information of the candidate hidden layer is calculated based on ordinary differential equations and converted into hidden state sequence output.

[0035] The gated recurrent unit adopts a structure similar to the existing GRU. The hidden state is calculated using the derivative of the hidden state function between observations. The GRU network model is continuous. By solving ordinary differential equations, the required hidden state sequence can be obtained. In each evaluation process, the equation is solved starting from the initial state h0.

[0036] This GRU network model mainly uses ordinary differential equations to calculate the output ht of the gated recurrent unit. Specifically, the ordinary differential equation can use the Euler method or the Runge-Kutta method. The model uses the function odeRNNCell to calculate the change or derivative of the hidden dynamics at time t. As shown in the following formula, odeRNNCell is a common initial value problem, which depends on the initial observation value y0 and the value of the hidden state h0 at time t0.

[0037] y t =ODESOLVER(odeRNNCell,tuple(y0,h0),t).

[0038] Among them, ODESOLVER is an ordinary differential solver, odeRUNCell is an ordinary differential recursive neuron function,

[0039] Reset Gate t , update gate z t , candidate hidden layer h t and output gate o t The update functions are:

[0040] r t =σ(W r x t +U r h t-1 +b r )

[0041] z t =σ(Wz x t +U z h t-1 +b r )

[0042]

[0043] o t =σ(W o h t +b0)

[0044] Among them, t is the time series, W r To reset the connection weight of the current input in the gate, U r To reset the connection weights of the previous layer in the gate, W z To update the connection weight of the current input in the gate, U z To update the connection weight of the previous layer in the gate, W o is the connection weight of the output, b r is the bias parameter of the reset gate, b o is the output deviation parameter.

[0045] The method is as follows:

[0046] First, bring the initial value and the initial value of the hidden layer into x and h.

[0047] x←states[0]

[0048] h←states[1];

[0049] Update the parameters of the GRU network model by deriving the above update function.

[0050]

[0051] According to the architecture of the GRU network model, the corresponding gradient of ht is solved.

[0052]

[0053] o t ←σ(W o h t +b o )

[0054]

[0055] This gradient value can be used to verify whether the code of our neural network calculation value is correct. Neural network models with more layers will have gradient disappearance and gradient explosion problems during training, which will cause the entire network to fail to function normally. Using gradient values ​​for verification can further improve the accuracy of neural network construction.

[0056] The GRU network model provided in the first aspect of the present invention is based on ordinary differential equations, takes the derivative of the hidden state as a parameter, and transforms the existing discrete type into a continuous space. During the training process, it does not need to be passed to each function in the forward propagation for evaluation in sequence, so that a large amount of space does not need to be consumed to store intermediate results, which greatly reduces the storage space and improves memory efficiency. In addition, during the evaluation or training process, by explicitly changing the accuracy of the numerical integration, the speed and accuracy of the model can be freely adjusted. For example, more time can be spent on training a high-precision model, and the accuracy can be reduced when evaluating the prediction to improve the response speed of the system.

[0057] The second aspect of the present invention provides a feature extraction method, which can be applied to electronic devices or a separate application. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and other electronic devices. The embodiments of the present application do not impose any restrictions on the specific type of electronic device. The feature extraction method can be applied to a variety of fields, such as face recognition, wound type recognition, bird recognition, plant type recognition, etc., and specific examples are not given here. Specifically, Figure 2 As shown, the feature extraction includes steps S01 to S04.

[0058] Step S01: Acquire at least one image, each of the at least one image including at least one target to be identified.

[0059] The image can be an image acquired in real time, or an image stored in the cloud or locally stored, and there is no limitation here. The image acquisition method can be based on an ordinary camera, a video camera or an RGB-D camera. Preferably, an RGB-D camera is used to acquire an RGB-D image, which includes an RGB image and a depth image. The Azure Kinect DK of the RGB-D camera is used to capture the depth image with the target to be identified. The Azure Kinect DK is equipped with a 1-megapixel advanced depth camera, a 360° microphone array, a 12-megapixel full HD camera and a direction sensor, which is suitable for advanced computer vision and speech model development. In this step, if wound type recognition is to be achieved, the image must have a wound to be identified; if face recognition is to be achieved, the image must have a face to be identified; other applications need to include the corresponding target to be identified.

[0060] Step S02: pre-process each image in the at least one image.

[0061] Specifically, the preprocessing method of this solution is: fill the holes in each image of the at least one image with a bilateral filtering algorithm to remove noise to make the depth image clearer. Bilateral filtering consists of two Gaussian kernel functions, which nonlinearly combines the spatial domain information and grayscale domain information of the depth image, overcomes the drawbacks of Gaussian filtering, and takes into account the image edge information during the depth restoration process, making the image edge clearer and smoother than Gaussian filtering.

[0062] Bilateral filtering is defined as:

[0063]

[0064]

[0065]

[0066]

[0067] ω=ω s ×ω r ,

[0068] Among them, ω s represents the spatial domain filter kernel, (i, j) represents the position of the current point, (m, n) represents the middle position of the filter kernel, σ s Expressed as the standard deviation in the spatial domain;

[0069] ω r represents the pixel domain filter kernel, f(i, j) represents the grayscale value of the current traversal point, f(m, n) represents the grayscale value of the middle position of the filter kernel, σ r Represents the standard deviation in pixel domain;

[0070] β represents the number of convolution kernels after traversing the convolution kernel. s With ω r The sum of products.

[0071] w represents the filter kernel. F(m,n) represents the grayscale value of the center point of the filter kernel. Ω represents the image space, i and j belong to Ω, indicating that they are points in the image, i represents the horizontal coordinate of the traversal point, and j represents the vertical coordinate of the traversal point. m represents the horizontal coordinate of the center point of the filter kernel, and n represents the vertical coordinate of the center point of the filter kernel.

[0072] Step S03, convolve and pool each of the at least one image. Through convolution, the brighter pixels in the output image can represent the edges in the injury image, thereby obtaining the edge information of the injury image. Through pooling, the robustness of the network is enhanced, and it has a certain anti-disturbance effect. Specifically, a convolutional neural network can be used to convolve and pool each of the at least one image.

[0073] Step S04: input the convolved and pooled image into the trained GRU network model to perform feature extraction on each of the at least one target to be identified. The GRU network model is a GRU network model based on ordinary differential equations as described in the first aspect and any possible embodiment thereof.

[0074] The GRU network model in this step needs to be trained in advance, and its training set is different depending on the target to be identified.

[0075] The extracted features can be used to further classify and identify the target to be identified, so as to further drive the intelligent devices at the back end. For example, when it comes to face recognition, the intelligent devices can be access control devices, payment terminals or others; when it comes to wound type recognition, they can identify knife wounds, abrasions and other types.

[0076] The feature extraction method of the second aspect of the present invention is based on the GRU network model in the first aspect, which parameterizes the hidden state as a continuous function of time, and automatically connects the parameters of nearby "network layers" together, thereby reducing the number of parameters required for supervised learning tasks, saving network training time, and ensuring the stability of the model; the entire method is based on RGB-D images, which comprehensively utilizes color and depth information and adopts a feature extraction strategy of multiple data modes to perform RGB-D object recognition.

[0077] The third aspect of the present invention discloses a feature extraction device, including a memory and a controller connected in sequence, wherein a computer program is stored on the memory, and the controller is used to read the computer program and execute a feature extraction method described in the second aspect and any one of the possibilities thereof. Specifically, the memory may include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a flash memory (Flash Memory), a first-in-first-out memory (FIFO) and a first-in-last-out memory (FILO), etc.; the controller may not be limited to a microprocessor of the STM32F105 series. In addition, the computer device may also include, but is not limited to, a power supply unit, a display screen and other necessary components.

[0078] The third aspect of the present invention discloses a feature extraction device, the operating principle of which is the same as that of the second aspect and will not be repeated here.

[0079] The fourth aspect of the present invention discloses a computer-readable storage medium, on which instructions are stored, and when the instructions are run on a computer, a feature extraction method described in the second aspect and any one of its possibilities is executed. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive and / or a memory stick, etc., and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0080] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A GRU network model based on ordinary differential equations, characterized in that: The invention comprises at least one gated recurrent unit GRU, each of the at least one GRU comprises a candidate hidden layer, the candidate hidden layer calculates input information of the candidate hidden layer based on an ordinary differential equation and converts it into a hidden state sequence output, and the gated recurrent unit comprises a reset gate r t , update gate z t and candidate hidden layers Reset Gate t , update gate z t , candidate hidden layer and output gate o t The update functions are: r t =σ(W r x t +U r h t-1 +b r ) z t =σ(W z x t +U z h t-1 +b r ) o t =σ(W o h t +b0) Among them, t is the time series, W r To reset the connection weight of the current input in the gate, U r To reset the connection weights of the previous layer in the gate, W z To update the connection weight of the current input in the gate, U z To update the connection weight of the previous layer in the gate, W o is the connection weight of the output, b r is the bias parameter of the reset gate, b o is the output deviation parameter, h t-1 The status information parameters of the previous moment.

2. The GRU network model based on ordinary differential equations according to claim 1, characterized in that: The candidate hidden layer uses the Euler method or the Runge-Kutta method to calculate the input information of the candidate hidden layer and convert it into a hidden state sequence output.

3. A feature extraction method, characterized in that: The following steps are involved: Acquire at least one image, each of the at least one image includes at least one target to be identified; perform convolution and pooling on each of the at least one image and input the convolution and pooling operations into a trained GRU network model to perform feature extraction on each of the at least one target to be identified, wherein the GRU network model is a GRU network model based on ordinary differential equations as described in any one of claims 1 or 2.

4. A feature extraction method according to claim 3, characterized in that: After acquiring at least one image, the method further includes: a step of preprocessing each of the at least one image.

5. A feature extraction method according to claim 4, characterized in that: The step of preprocessing each of the at least one image includes: filling holes in each of the at least one image using a bilateral filtering algorithm.

6. A feature extraction method according to claim 3, characterized in that: The image is an RGB-D image.

7. A feature extraction method according to claim 3, characterized in that: A convolutional neural network is used to perform convolution and pooling on each of the at least one image.

8. A feature extraction device, comprising a memory and a controller which are communicatively connected in sequence, wherein a computer program is stored in the memory, characterized in that: The controller is used to read the computer program and execute a feature extraction method as described in any one of claims 3-7.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed on a computer, a feature extraction method as described in any one of claims 3 to 7 is executed.