Optical Flow Estimation Method, Device and System Based on Global Pulse Aggregation

Abstract

The present invention discloses an optical flow estimation method, device and system based on global pulse aggregation. The optical flow estimation method includes obtaining a pulse image to be measured; inputting the pulse image to be measured into a pre-trained optical flow estimation network to obtain a predicted optical flow; the optical flow estimation network includes a feature extraction module, a feature matching module and a cyclic decoding module; the feature extraction module is used to clip the received pulse image into several pulse sub-streams, perform feature extraction on each pulse sub-stream, and generate corresponding context features; the feature matching module is used to match the context features with the features obtained by performing a convolution operation on the pulse image at the previous moment, and encode the matching results into motion features with corresponding flow fields; the cyclic decoding module is used to perform cyclic decoding based on the context features and motion features to obtain optical flow. The present invention can extract context features and motion features containing global temporal information from a continuous pulse stream, and then learn to obtain optical flow.

Description

Technical Field

[0001] The present invention belongs to the field of optical flow estimation, and in particular relates to an optical flow estimation method, device and system based on global pulse aggregation. Background Art

[0002] Optical flow estimation is an important direction in computer vision research and plays an important role in target tracking, image enhancement, action recognition, etc. At present, deep learning technology has achieved success in optical flow estimation due to its powerful feature representation ability. However, in extreme environments or high-speed motion scenes, traditional cameras are prone to losing or missing targets due to motion blur due to their low-frequency imaging characteristics. In order to capture the motion characteristics in high-speed scenes, pulse cameras have become a hot topic of current research, providing a new perspective for optical flow estimation in high-speed scenes. However, previous studies only focused on the motion solution problem between two pulse streams, which ignored the temporal information in the globally aggregated pulse stream, thereby reducing the model's ability to express pulse stream features in complex scenes. Summary of the Invention

[0003] To address the above problems, the present invention proposes an optical flow estimation method, device and system based on global pulse aggregation, which can extract contextual features and motion features containing global temporal information from continuous pulse streams, and then learn the optical flow.

[0004] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0005] In a first aspect, the present invention provides an optical flow estimation method based on global pulse aggregation, comprising:

[0006] Acquire the pulse image to be measured;

[0007] The pulse image to be tested is input into a pre-trained optical flow estimation network to obtain a predicted optical flow; the optical flow estimation network includes a feature extraction module, a feature matching module and a cyclic decoding module; the feature extraction module is used to clip the received pulse image into several pulse sub-streams, perform feature extraction on each pulse sub-stream, and generate corresponding context features; the feature matching module is used to match the context features with the features obtained by performing a convolution operation on the pulse image at the previous moment, and encode the matching results into motion features with corresponding flow fields; the cyclic decoding module is used to perform cyclic decoding based on the context features and motion features to obtain optical flow.

[0008] Optionally, the feature extraction module includes:

[0009] A global pulse information aggregation module is used to calculate the global pixel matrix G based on the received pulse sub-streams;

[0010] A timing information aggregation module is used to calculate a timing pixel matrix A based on the received pulse sub-stream;

[0011] A maximum pulse information aggregation module is used to calculate a maximum pulse pixel matrix M based on the maximum value of each pixel point based on the received pulse sub-stream;

[0012] The convolution module is connected to the global pulse information aggregation module, the temporal information aggregation module and the maximum pulse information aggregation module respectively, and calculates the context features based on the global pixel matrix G, the temporal pixel matrix A and the maximum pulse pixel matrix M.

[0013] Optionally, the calculation formula of the global pixel matrix G is:

[0014]

[0015] The calculation formula of the temporal pixel matrix A is:

[0016]

[0017] The calculation formula of the maximum pulse pixel matrix M is:

[0018]

[0019] Among them, g(i,j) represents the global pulse information of the pulse sub-stream at the pixel point (i,j), is the global pulse information aggregation transformation, s(x,t) represents the pulse sub-stream, Represents the information of pixel (i, j) on the kth image, N represents the length of the pulse sub-stream, It is a temporal information aggregation transformation, δ2(x, t) and δ1(x, t) represent the emission time of the next and previous pulse sub-streams at the time and space time (x, t), x represents a pixel set, t represents a certain moment, ||·||2 represents L2 regularization, It is the maximum pulse information aggregation transformation, m(i, j) represents the maximum pulse information of the pulse sub-stream at the pixel point (i, j).

[0020] Optionally, the feature matching module includes: a correlation calculation module, a first convolution operation block, a second convolution operation block, a first feature stacking block, a third convolution operation block, a fourth convolution operation block, a fifth convolution operation block and a second feature stacking block;

[0021] The relevance calculation module calculates the relevance based on the received context features;

[0022] The first convolution operation block and the second convolution operation block are sequentially arranged to perform convolution on the output data of the correlation calculation module;

[0023] The third convolution operation block and the fourth convolution operation block are sequentially arranged to perform convolution on the pulse image at the previous moment;

[0024] The first feature stacking block, the fifth convolution operation block and the second feature stacking block are arranged in sequence. The first feature stacking block stacks the features output by the second convolution operation block and the fourth convolution operation block, and then the fifth convolution operation block convolves the stacked features again. After obtaining the features, they are stacked with the pulse image at the previous moment to obtain motion features.

[0025] Optionally, the correlation between features is calculated as follows:

[0026]

[0027] Among them, C i (x,y,x',y') represents the correlation between the context feature corresponding to the sampling time i and the feature obtained by convolution operation on the pulse image at the previous moment, (x,y) is the context feature The coordinates of (x', y') are the features obtained by convolution operation on the pulse image at the previous moment. Coordinates, T represents transposition, and the size of a single pulse image is H×W.

[0028] Optionally, the cyclic decoding module includes: a third feature stacking block, a first convolutional gated cyclic unit, a second convolutional gated cyclic unit, N prediction heads, a fourth feature stacking block and an upsampling block;

[0029] The third feature stacking block stacks the received context features and motion features, and inputs the stacked features into the first convolutional gated recurrent unit and the second convolutional gated recurrent unit respectively to obtain the hidden state h t+1 ;

[0030] The hidden state h t+1 Each prediction head is sent to each prediction head respectively, and each prediction head is a 3×3 convolution to obtain the initial flow

[0031] The initial flow is processed using the fourth feature stacking block Stack and upsample using the upsampling module to obtain the final optical flow O1~O N .

[0032] Optionally, the mathematical expressions of the first convolutional gated recurrent unit and the second convolutional gated recurrent unit are:

[0033] z t =σ(Conv 3×3 ([ht-1 ,x t ],W z ))

[0034] r t =σ(Conv 3×3 ([h t-1 ,x t ],W r ))

[0035] h t =tanh(Conv 3×3 ([r t ⊙h t-1 ,x t ],W h ))

[0036] h t+1 =(1-z t )⊙h t-1 +z t ⊙h t

[0037] Among them, x t is the feature after stacking motion features and context features, h t-1 is the implicit feature at time t-1, σ is the sigmoid function, ⊙ is the element-wise multiplication, W z 、W r 、W h All represent the corresponding weights, Conv 3×3 represents 3×3 convolution, z t and r t They represent the update gate and reset gate respectively. The update gate is used to control the extent to which the state information of the previous moment is brought into the current state. The larger the value of the update gate, the more state information of the previous moment is brought in. The reset gate controls how much information of the previous state is written into the current candidate set h. t The smaller the value of the reset gate is, the less information of the previous state is written, h t is the candidate hidden feature, h t is the final hidden feature, h t+1 is the implicit feature at time t+1.

[0038] Optionally, during the training process of the optical flow estimation network, a loss function is used to determine whether to stop training. The expression of the loss function is:

[0039]

[0040] Among them, O i is the true value flow field The corresponding predicted optical flow value, ||·||1 indicates L1 regularization.

[0041] In a second aspect, the present invention provides an optical flow estimation device based on global pulse aggregation, comprising:

[0042] An acquisition module, used for acquiring an image of a pulse to be measured;

[0043] An estimation module is used to input the pulse image to be tested into a pre-trained optical flow estimation network to obtain a predicted optical flow; the optical flow estimation network includes a feature extraction module, a feature matching module and a cyclic decoding module; the feature extraction module is used to clip the received pulse image into several pulse sub-streams, perform feature extraction on each pulse sub-stream, and generate corresponding context features; the feature matching module is used to match the context features with the features obtained by performing a convolution operation on the pulse image at the previous moment, and encode the matching results into motion features with corresponding flow fields; the cyclic decoding module is used to perform cyclic decoding based on the context features and motion features to obtain optical flow.

[0044] In a third aspect, the present invention provides an optical flow estimation system based on global pulse aggregation, comprising a storage medium and a processor;

[0045] The storage medium is used to store instructions;

[0046] The processor is configured to operate according to the instructions to execute the method according to any one of the first aspects.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention proposes an optical flow estimation method, device and system based on global pulse aggregation, which can extract contextual features and motion features containing global temporal information from continuous pulse streams, and then learn to obtain optical flow.

[0049] The present invention proposes a new optical flow estimation network that combines a global pulse aggregation module with optical flow estimation. The binary pulse stream is converted through a timing information aggregation module to better represent the integration process of each pulse stream. At the same time, the pulse stream is averaged along the time axis, the timing information in the pulse stream is globally aggregated, and the extracted pulse stream features are matched to obtain motion features. In addition, the residual of the flow field is estimated by contextual features and motion features, thereby optimizing the flow field, and finally the flow field is cyclically decoded. Through the above settings, the network can extract global timing information from the continuous pulse stream and then learn to obtain optical flow. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which:

[0051] Figure 1 Schematic diagram of the process of an optical flow estimation method based on global pulse aggregation according to an embodiment of the present invention;

[0052] Figure 2 This is a diagram of a feature extraction and context extraction network structure according to an embodiment of the present invention;

[0053] Figure 3 This is a feature matching structure diagram of an embodiment of the present invention;

[0054] Figure 4 This is a structural diagram of a cyclic decoding module according to an embodiment of the present invention;

[0055] Figure 5 This is a diagram of the overall network structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features specified as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0058] Example 1

[0059] An embodiment of the present invention provides an optical flow estimation method based on global pulse aggregation, comprising the following steps:

[0060] (1) Obtaining the pulse image to be measured;

[0061] (2) The pulse image to be tested is input into a pre-trained optical flow estimation network to obtain a predicted optical flow; the optical flow estimation network includes a feature extraction module, a feature matching module and a cyclic decoding module; the feature extraction module is used to clip the received pulse image into several pulse sub-streams, perform feature extraction on each pulse sub-stream, and generate corresponding context features; the feature matching module is used to match the context features with the features obtained by performing a convolution operation on the pulse image at the previous moment, and encode the matching results into motion features with corresponding flow fields; the cyclic decoding module is used to perform cyclic decoding based on the context features and motion features to obtain optical flow.

[0062] In a specific implementation of the embodiment of the present invention, Figure 2 and 5 As shown, the feature extraction module includes:

[0063] A global pulse information aggregation module is used to calculate the global pixel matrix G based on the received pulse sub-streams;

[0064] A timing information aggregation module is used to calculate a timing pixel matrix A based on the received pulse sub-stream;

[0065] A maximum pulse information aggregation module is used to calculate a maximum pulse pixel matrix M based on the maximum value of each pixel point based on the received pulse sub-stream;

[0066] The convolution module is connected to the global pulse information aggregation module, the temporal information aggregation module and the maximum pulse information aggregation module respectively, and calculates the context features based on the global pixel matrix G, the temporal pixel matrix A and the maximum pulse pixel matrix M.

[0067] The calculation formula of the global pixel matrix G is:

[0068]

[0069] The calculation formula of the temporal pixel matrix A is:

[0070]

[0071] The calculation formula of the maximum pulse pixel matrix M is:

[0072]

[0073] Among them, g(i,j) represents the global pulse information of the pulse sub-stream at the pixel point (i,j), is the global pulse information aggregation transformation, s(x,t) represents the pulse sub-stream, Represents the information of pixel (i, j) on the kth image, N represents the length of the pulse sub-stream, It is a temporal information aggregation transformation, δ2(x, t) and δ1(x, t) represent the emission time of the next and previous pulse sub-streams at the time and space time (x, t), x represents a pixel set, t represents a certain moment, ||·||2 represents L2 regularization, It is the global pulse information aggregation transformation, and m(i, j) represents the maximum pulse information of the pulse sub-stream at the pixel point (i, j).

[0074] In a specific implementation of the embodiment of the present invention, Figure 3 As shown, the feature matching module includes: a correlation calculation module, a first convolution operation block, a second convolution operation block, a first feature stacking block, a third convolution operation block, a fourth convolution operation block, a fifth convolution operation block and a second feature stacking block; in a specific implementation process, the first convolution operation block uses a 1*1 convolution kernel; the second convolution operation block uses a 3*3 convolution kernel; the third convolution operation block uses a 7*7 convolution kernel; the second convolution operation block uses a 3*3 convolution kernel; the fifth convolution operation block uses a 3*3 convolution kernel;

[0075] The relevance calculation module calculates the relevance based on the received context features;

[0076] The first convolution operation block and the second convolution operation block are sequentially arranged to perform convolution on the output data of the correlation calculation module;

[0077] The third convolution operation block and the fourth convolution operation block are sequentially arranged to perform convolution on the pulse image at the previous moment;

[0078] The first feature stacking block, the fifth convolution operation block, and the second feature stacking block are arranged in sequence. The first feature stacking block stacks the features output by the second convolution operation block and the fourth convolution operation block, and then the fifth convolution operation block convolves the stacked features again to obtain the features and then stacks them with the pulse image at the previous moment to obtain motion features. Specifically, Figure 3 If we define the pulse sub-flow S0~S N The corresponding flow rate is O1~O N ,So Figure 3 The input in the upper left corner is the correlation C1~C N The current estimated flow is actually the flow at the previous moment, defined as Then the red input is The default value is 0. And so on. Then perform the convolution operation. The features from the two paths are stacked (note that this means C iThe corresponding features are i The corresponding features are stacked, so N stacking is performed), and then the stacked features are convolved again, and the features are stacked with the current estimated flow to obtain motion features, such as defined as U0~U N Then all motion features and context features are stacked to get the output, for example, defined as

[0079] The calculation formula for the correlation between features is:

[0080]

[0081] Among them, C i (x,y,x',y') represents the correlation between the context feature corresponding to the sampling time i and the feature obtained by convolution operation on the pulse image at the previous moment, (x,y) is the context feature The coordinates of (x', y') are the features obtained by convolution operation on the pulse image at the previous moment. Coordinates, T represents transposition, and the size of a single pulse image is H×W.

[0082] In a specific implementation of the embodiment of the present invention, Figure 5 As shown, the cyclic decoding module includes: a third feature stacking block, a first convolutional gated cyclic unit, a second convolutional gated cyclic unit, N prediction heads, a fourth feature stacking block and an upsampling block;

[0083] The third feature stacking block stacks the received context features and motion features, and inputs the stacked features into the first convolutional gated recurrent unit and the second convolutional gated recurrent unit respectively to obtain the hidden state h t+1 ;

[0084] The hidden state h t+1 Each prediction head is sent to each prediction head respectively, and each prediction head is a 3×3 convolution to obtain the initial flow

[0085] The initial flow is processed using the fourth feature stacking block Stack and upsample using the upsampling module to obtain the final optical flow O1~O N . Specifically, Figure 4 The input is Figure 3 Output After two convolutional recurrent gated units, the hidden state h is obtained t+1 . Then the hidden state h t+1 Send them to prediction heads 1 to N respectively (each prediction head is a 3×3 convolution) to get the initial flow Then Stack and then upsample the whole to get the final optical flow O1~O N .

[0086] The mathematical expressions of the first convolutional gated recurrent unit and the second convolutional gated recurrent unit are:

[0087] z t =σ(Conv 3×3 ([h t-1 ,x t ],W z ))

[0088] r t =σ(Conv 3×3 ([h t-1 ,x t ],W r ))

[0089] h t =tanh(Conv 3×3 ([r t ⊙h t-1 ,x t ],W h ))

[0090] h t+1 =(1-z t )⊙h t-1 +z t ⊙h t

[0091] Among them, x t is the feature after stacking motion features and context features, h t-1 is the implicit feature at time t-1, σ is the sigmoid function, ⊙ is the element-wise multiplication, W z 、W r 、W h All represent the corresponding weights, Conv 3×3 represents 3×3 convolution, z t and r t They represent the update gate and reset gate respectively. The update gate is used to control the extent to which the state information of the previous moment is brought into the current state. The larger the value of the update gate, the more state information of the previous moment is brought in. The reset gate controls how much information of the previous state is written into the current candidate set h. t The smaller the value of the reset gate is, the less information of the previous state is written, h t is the candidate hidden feature, h t is the final hidden feature, h t+1 is the implicit feature at time t+1.

[0092] In a specific implementation of the embodiment of the present invention, during the training process of the optical flow estimation network, a loss function is used to determine whether to stop training. The expression of the loss function is:

[0093]

[0094] Among them, O i is the true value flow field The corresponding predicted optical flow value, ||·||1 indicates L1 regularization.

[0095] The method in the embodiment of the present invention is described in detail below with reference to a specific implementation manner.

[0096] like Figure 1 As shown, the method includes:

[0097] S1. Prepare dataset;

[0098] Prepare continuous pulse images for optical flow network training, namely pulse stream S(x,t).

[0099] S2. Feature extraction.

[0100] Clip the input pulse stream S(x,t) into pulse sub-streams s(x,t) and extract the features of the pulse sub-streams. Specifically:

[0101] Construct a temporal information aggregation module. Each pulse in the pulse stream represents the result of the photon integration process, rather than the current state. Here, when the photon accumulation reaches a certain threshold, it is recorded as "1", and when it does not reach the threshold, it is recorded as "0". Therefore, using features extracted from binary pulses for matching may not be suitable for reflecting the structure of the scene. It is recommended to use emission time to represent the information contained in the pulse. This can better represent the rate at which photons arrive at each pixel, thereby aggregating stage-by-stage temporal information.

[0102] If the timing information of the pulse substream s(x,t) is represented as A(x,t), the stage-by-stage timing information aggregation can be expressed as:

[0103]

[0104] in, It is a temporal information aggregation transformation. δ2(x, t) and δ1(x, t) represent the emission time of the next and previous pulse substreams at the spatiotemporal moment (x, t), where x represents a set of pixels and t represents a certain moment. ||·||2 represents L2 regularization. The temporal information aggregation module is used in the present invention to reflect the emission rate of the pulse. The firing rate here is a statistical concept, and the average value of the individual firing rate interval is related to the firing rate. The motion in the scene changes the brightness, and the Poisson process of the arrival of photons causes fluctuations in the emission rate. Therefore, the emission rate is time-varying at each pixel. This module can better restore the dynamic process of the pulse, rather than obtaining a more constant emission rate at a longer time interval.

[0105] At the same time, a global pulse information aggregation module is constructed to aggregate the global timing information in the pulse stream by averaging the pulse stream along the time axis. If the global pulse information aggregation information of the pulse substream s(x,t) is represented as G, g(i,j)∈G, then the global pulse information aggregation information can be expressed as:

[0106]

[0107] in, is the global pulse information aggregation transformation. g(i,j) represents the global pulse information of the pulse substream at pixel (i,j). Represents the information of point (i, j) on the kth image, and N represents the length of the pulse sub-stream.

[0108] Then, the maximum pulse information aggregation of each pixel on the pulse substream is obtained to obtain the pixel matrix M, m(i, j)∈M. Let the number of "0"s at the pixel point (i, j) of the pulse substream s(x, t) along the time axis be m, and the number of "1"s be n, then:

[0109]

[0110] in, is the maximum impulse information aggregation transform. m(i, j) represents the maximum impulse information of the impulse substream at point (i, j). Subsequently, the corresponding convolution transform operations (using a 3×3 convolution kernel) are performed on A, G, and M respectively to extract the characteristics of the impulse substream.

[0111] S3. Feature matching;

[0112] In order to extract the continuous motion process of the optical scene in the pulse, feature matching is performed on the correlation between the pulse flow features and encoded into motion features with corresponding flow fields. For each sampling time i, the correlation between the features is constructed:

[0113]

[0114] Among them, C i (x,y,x',y') represents the correlation between the context feature corresponding to the sampling time i and the feature obtained by convolution operation on the pulse image at the previous moment, (x,y) is the context feature The coordinates of (x', y') are the features obtained by convolution operation on the pulse image at the previous moment. The coordinates of the image are shown in Figure 2, T represents transposition, and the size of a single pulse image is H × W. The larger the value of C, the higher the similarity. Conversely, the smaller the value of C, the lower the similarity.

[0115] S4. Loop decoding

[0116] A recurrent decoding module is constructed. In each iteration, the recurrent decoding module estimates the residual of the flow field using motion features and context features, thereby optimizing the flow field. A single hidden state is used for all estimated flow fields. Different prediction heads are used to estimate the residuals of different flow fields. To further refine the flow field, an update operation is introduced to update the original flow field. The update operation estimates the flow sequence from the initial starting point. Its core component is the gated recurrent unit based on the GRU unit:

[0117] z t =σ(Conv 3×3 ([h t-1 ,x t ],W z )) (5)

[0118] r t =σ(Conv 3×3 ([h t-1 ,x t ],W r )) (6)

[0119] h t =tanh(Conv 3×3 ([r t ⊙h t-1 ,x t ],W h )) (7)

[0120] h t+1 =(1-z t )⊙h t-1 +z t ⊙h t (8)

[0121] Among them, x t is the input feature, i.e., the stacking of motion features and context features. t-1is the implicit feature at time t-1, which contains the relevant information of the previous node. σ is the sigmoid function, which can transform the data into a value in the range of 0-1, thus acting as a gating signal. ⊙ is the element-wise multiplication, and W represents the corresponding weight. Conv 3×3 represents 3×3 convolution, z t and r t Represent the update gate and reset gate respectively. t is the candidate hidden feature, h t The update gate is used to control the extent to which the state information of the previous moment is brought into the current state. The larger the value of the update gate, the more state information of the previous moment is brought in. The reset gate controls how much information of the previous state is written into the current candidate set h. t The smaller the value of the reset gate is, the less information of the previous state is written.

[0122] The optical flow is learned by constructing a flow prediction head using 3×3 convolution. There are N flow prediction heads in total, each of which outputs the flow field between frames of the corresponding pulse flow sequence.

[0123] The true flow field between two frames in a given pulse sub-flow The loss function is defined as:

[0124]

[0125] Among them, O i is the true value flow field The corresponding predicted optical flow value, ||·||1 indicates L1 regularization.

[0126] S5. Network Training

[0127] Train each module and connect them to each other, fix the weights of extracting optical flow features, perform cyclic decoding training, and obtain the optical flow estimation network.

[0128] S6. Network test

[0129] The loss function is used to determine whether to stop training. The expression of the loss function is:

[0130]

[0131] Among them, O i is the true value flow field The corresponding predicted optical flow value, ||·||1 indicates L1 regularization.

[0132] S7. Practical Application

[0133] The trained optical flow estimation network is used to perform forward calculations on the test image pairs, and the output is the predicted optical flow.

[0134] Example 2

[0135] Based on the same inventive concept as that of Example 1, an embodiment of the present invention provides an optical flow estimation device based on global pulse aggregation, including:

[0136] An acquisition module, used for acquiring an image of a pulse to be measured;

[0137] An estimation module is used to input the pulse image to be tested into a pre-trained optical flow estimation network to obtain a predicted optical flow; the optical flow estimation network includes a feature extraction module, a feature matching module and a cyclic decoding module; the feature extraction module is used to clip the received pulse image into several pulse sub-streams, perform feature extraction on each pulse sub-stream, and generate corresponding context features; the feature matching module is used to match the context features with the features obtained by performing a convolution operation on the pulse image at the previous moment, and encode the matching results into motion features with corresponding flow fields; the cyclic decoding module is used to perform cyclic decoding based on the context features and motion features to obtain optical flow.

[0138] Example 3

[0139] Based on the same inventive concept as that of Example 1, an embodiment of the present invention provides an optical flow estimation system based on global pulse aggregation, including a storage medium and a processor;

[0140] The storage medium is used to store instructions;

[0141] The processor is configured to operate according to the instructions to perform the method according to any one of the embodiments 1.

[0142] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0143] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0146] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which are all protected by the present invention.

[0147] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. An optical flow estimation method based on global pulse aggregation, characterized in that: include: Acquire the pulse image to be measured; Inputting the pulse image to be tested into a pre-trained optical flow estimation network to obtain a predicted optical flow; The optical flow estimation network includes a feature extraction module, a feature matching module and a loop decoding module; The feature extraction module is used to clip the received pulse image into several pulse sub-streams, extract features from each pulse sub-stream respectively, and generate corresponding context features; the feature matching module is used to match the context features with the features obtained by performing a convolution operation on the pulse image at the previous moment, and encode the matching results into motion features with corresponding flow fields; the cyclic decoding module is used to perform cyclic decoding based on the context features and motion features to obtain optical flow.

2. The optical flow estimation method based on global pulse aggregation according to claim 1, characterized in that: The feature extraction module includes: A global pulse information aggregation module is used to calculate the global pixel matrix G based on the received pulse sub-streams; A timing information aggregation module is used to calculate a timing pixel matrix A based on the received pulse sub-stream; A maximum pulse information aggregation module is used to calculate a maximum pulse pixel matrix M based on the maximum value of each pixel point based on the received pulse sub-stream; The convolution module is connected to the global pulse information aggregation module, the temporal information aggregation module and the maximum pulse information aggregation module respectively, and calculates the context features based on the global pixel matrix G, the temporal pixel matrix A and the maximum pulse pixel matrix M.

3. The optical flow estimation method based on global pulse aggregation according to claim 2, characterized in that: The calculation formula of the global pixel matrix G is: The calculation formula of the temporal pixel matrix A is: The calculation formula of the maximum pulse pixel matrix M is: Among them, g(i,j) represents the global pulse information of the pulse sub-stream at the pixel point (i,j), is the global pulse information aggregation transformation, s(x,t) represents the pulse sub-stream, Represents the information of pixel (i, j) on the kth image, N represents the length of the pulse sub-stream, It is a temporal information aggregation transformation, δ2(x, t) and δ1(x, t) represent the emission time of the next and previous pulse sub-streams at the time and space time (x, t), x represents a pixel set, t represents a certain moment, ||·||2 represents L2 regularization, It is the maximum pulse information aggregation transformation, m(i, j) represents the maximum pulse information of the pulse sub-stream at the pixel point (i, j).

4. The optical flow estimation method based on global pulse aggregation according to claim 1, characterized in that: The feature matching module includes: a correlation calculation module, a first convolution operation block, a second convolution operation block, a first feature stacking block, a third convolution operation block, a fourth convolution operation block, a fifth convolution operation block and a second feature stacking block; The relevance calculation module calculates the relevance based on the received context features; The first convolution operation block and the second convolution operation block are sequentially arranged to perform convolution on the output data of the correlation calculation module; The third convolution operation block and the fourth convolution operation block are arranged in sequence to perform convolution on the pulse image at the previous moment; the first feature stacking block, the fifth convolution operation block and the second feature stacking block are arranged in sequence, the first feature stacking block stacks the features output by the second convolution operation block and the fourth convolution operation block, and then the fifth convolution operation block convolves the stacked features again, and after obtaining the features, they are stacked with the pulse image at the previous moment to obtain motion features.

5. The optical flow estimation method based on global pulse aggregation according to claim 4, characterized in that: The calculation formula for the correlation between features is: Among them, C i (x,y,x',y') represents the correlation between the context feature corresponding to the sampling time i and the feature obtained by convolution operation on the pulse image at the previous moment, (x,y) is the context feature The coordinates of (x', y') are the features obtained by convolution operation on the pulse image at the previous moment. Coordinates, T represents transposition, and the size of a single pulse image is H×W.

6. The optical flow estimation method based on global pulse aggregation according to claim 1, characterized in that: The cyclic decoding module includes: a third feature stacking block, a first convolutional gated cyclic unit, a second convolutional gated cyclic unit, N prediction heads, a fourth feature stacking block and an upsampling block; The third feature stacking block stacks the received context features and motion features, and inputs the stacked features into the first convolutional gated recurrent unit and the second convolutional gated recurrent unit respectively to obtain the hidden state h t+1 ; The hidden state h t+1 Each prediction head is sent to each prediction head respectively, and each prediction head is a 3×3 convolution to obtain the initial flow The initial flow is processed using the fourth feature stacking block Stack and upsample using the upsampling module to obtain the final optical flow O1~O N .

7. The optical flow estimation method based on global pulse aggregation according to claim 6, characterized in that: The mathematical expressions of the first convolutional gated recurrent unit and the second convolutional gated recurrent unit are: z t =σ(Conv 3×3 ([h t-1 ,x t ],W z )) r t =σ(Conv 3×3 ([h t-1 ,x t ],W r )) h t =tanh(Conv 3×3 ([r t ⊙h t-1 ,x t ],W h )) h t+1 =(1-z t )⊙h t-1 +z t ⊙h t Among them, x t is the feature after stacking motion features and context features, h t-1 is the implicit feature at time t-1, σ is the sigmoid function, ⊙ is the element-wise multiplication, W z 、W r 、W h All represent the corresponding weights, Conv 3×3 represents 3×3 convolution, z t and r t They represent the update gate and reset gate respectively. The update gate is used to control the extent to which the state information of the previous moment is brought into the current state. The larger the value of the update gate, the more state information of the previous moment is brought in. The reset gate controls how much information of the previous state is written into the current candidate set h. t The smaller the value of the reset gate is, the less information of the previous state is written, h t is the candidate hidden feature, h t is the final hidden feature, h t+1 is the implicit feature at time t+1.

8. The optical flow estimation method based on global pulse aggregation according to claim 1, characterized in that: During the training process of the optical flow estimation network, a loss function is used to determine whether to stop training. The expression of the loss function is: Among them, O i is the true value flow field The corresponding predicted optical flow value, ||·||1 indicates L1 regularization.

9. An optical flow estimation device based on global pulse aggregation, characterized in that: include: An acquisition module, used for acquiring an image of a pulse to be measured; An estimation module, configured to input the pulse image to be tested into a pre-trained optical flow estimation network to obtain a predicted optical flow; The optical flow estimation network includes a feature extraction module, a feature matching module and a loop decoding module; The feature extraction module is used to clip the received pulse image into several pulse sub-streams, extract features from each pulse sub-stream respectively, and generate corresponding context features; the feature matching module is used to match the context features with the features obtained by performing a convolution operation on the pulse image at the previous moment, and encode the matching results into motion features with corresponding flow fields; the cyclic decoding module is used to perform cyclic decoding based on the context features and motion features to obtain optical flow.

10. An optical flow estimation system based on global pulse aggregation, characterized in that: including storage media and processors; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the method according to any one of claims 1-8.

Images