A neural network optical flow estimation method, device and medium implemented by FPGA

Abstract

The present invention discloses a neural network optical flow estimation method, device and medium implemented by FPGA, the method comprising: obtaining two consecutive frames of pictures, performing preprocessing according to the pictures to obtain image blocks; sampling according to the image blocks to obtain three-layer pyramid blocks; performing feature extraction using a binary neural network according to the three-layer pyramid blocks; obtaining a feature pyramid according to the result of the feature extraction; and outputting optical flow information according to the feature pyramid. The present invention is based on hierarchical processing and feature extraction of images, simplifies the calculation process and realizes high-speed optical flow estimation. At the same time, the method can conveniently realize end-to-end pipeline design and improve real-time performance. At the same time, it has passed simulation and synthesis, and can be actually deployed in an ALTERA FPGA, meeting the real-time requirements of most real-time systems, and can be widely used in the field of image processing technology.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a neural network optical flow estimation method, device and medium implemented by FPGA. Background Art

[0002] Optical flow is the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane. It uses the change of pixels in the time domain in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame. Optical flow estimation is one of the most important technologies for motion analysis. It can recover the motion information of the target object and the background from adjacent images, thereby realizing target detection, motion tracking and feature recognition. At present, optical flow calculation technology can be divided into differential method, region-based matching method, energy-based method, phase-based method and neurodynamic method according to the basic principle. The differential method uses the spatiotemporal differential (i.e., spatiotemporal gradient function) of the time-varying image grayscale (or its filtered form) to calculate the velocity vector of the pixel. The region-based method first locates similar regions and then calculates the optical flow through the displacement of similar regions. The energy-based method uses the frequency information output by the filter group with adjustable speed for calculation. The phase-based method uses the phase information for optical flow calculation. The neurodynamic method is a neurodynamic model of visual motion perception established by neural network.

[0003] Nowadays, the implementation scenarios that require optical flow estimation technology require not only high accuracy but also real-time performance, but the high-precision traditional methods implemented on high-performance computers are difficult to achieve the goal of real-time performance. Therefore, many research experts and scholars choose to deploy simplified methods on FPGAs, and most of them use the Lucas–Kanade optical flow algorithm, but this method has low accuracy and is difficult to use. Therefore, the current optical flow calculation technology either has a huge amount of calculation, consumes a large number of computing units and has low real-time performance, or has low accuracy and low practicality. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a neural network optical flow estimation method, device and medium implemented by FPGA, which can achieve high-efficiency and high-precision optical flow estimation and meet the real-time requirements of most real-time systems.

[0005] A first aspect of an embodiment of the present invention provides a neural network optical flow estimation method implemented by FPGA, comprising:

[0006] Obtain two consecutive frames of pictures, and perform preprocessing on the pictures to obtain image blocks;

[0007] According to the image block, sampling is performed to obtain a three-layer pyramid block;

[0008] According to the three-layer pyramid block, a binary neural network is used to extract features;

[0009] According to the result of the feature extraction, a feature pyramid is obtained;

[0010] Optical flow information is output according to the feature pyramid.

[0011] Optionally, acquiring two consecutive frames of pictures and performing preprocessing on the pictures to obtain image blocks includes:

[0012] In the same clock cycle, pixels of the image are acquired one by one;

[0013] Delay processing is performed on the pixels to obtain the image block.

[0014] Optionally, sampling the image block to obtain a three-layer pyramid block includes:

[0015] Downsampling the image block to obtain a downsampled image block;

[0016] Performing original resolution sampling on the image block to obtain an original image block;

[0017] Upsampling is performed on the image block to obtain an upsampled image block.

[0018] Optionally, the extracting features using a binary neural network according to the three-layer pyramid block includes:

[0019] Converting the three-layer pyramid block from a parallel data stream into a serial data stream;

[0020] According to the serial data stream, time-division feature extraction is performed using the time-division multiplexed binary neural network.

[0021] Optionally, the time-sharing feature extraction using the binary neural network with time-sharing multiplexing includes:

[0022] The feature extraction of the down-sampled image block is multiplexed once, the feature extraction of the original image block is multiplexed once, and the feature extraction of the up-sampled image block is multiplexed four times.

[0023] Optionally, outputting optical flow information according to the feature pyramid includes:

[0024] Perform a block matching operation according to the feature pyramid to obtain a first optical flow vector;

[0025] A smoothing operation is performed on the first optical flow vector to obtain a second optical flow vector.

[0026] Optionally, outputting optical flow information according to the feature pyramid further includes:

[0027] Performing a thinning operation on the second optical flow vector, and performing median filtering on a result of the thinning operation to obtain an optical flow map;

[0028] According to a preset period and the optical flow map, the optical flow vector of each pixel in the optical flow map is output one by one.

[0029] A second aspect of an embodiment of the present invention provides a neural network optical flow estimation device implemented by FPGA, comprising:

[0030] The first module is used to obtain two consecutive frames of pictures and perform preprocessing on the pictures to obtain image blocks;

[0031] The second module is used to sample and obtain a three-layer pyramid block according to the image block;

[0032] A third module is used to extract features using a binary neural network according to the three-layer pyramid block;

[0033] The fourth module is used to obtain a feature pyramid according to the result of the feature extraction;

[0034] The fifth module is used to output optical flow information according to the feature pyramid.

[0035] A third aspect of an embodiment of the present invention provides an electronic device, including a processor and a memory;

[0036] The memory is used to store programs;

[0037] The processor executes the program to implement the method described above.

[0038] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0039] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0040] The embodiment of the present invention first obtains two consecutive frames of pictures, performs preprocessing on the pictures to obtain image blocks; then, based on the image blocks, samples to obtain three-layer pyramid blocks; then, based on the three-layer pyramid blocks, uses a binary neural network to extract features; then, based on the results of the feature extraction, obtains a feature pyramid; and finally, outputs optical flow information based on the feature pyramid. Based on the hierarchical processing and feature extraction of images, the present invention simplifies the calculation process and achieves high-speed optical flow estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A schematic diagram of a flow chart of a neural network optical flow estimation method implemented by FPGA according to an embodiment of the present invention;

[0043] Figure 2 A block matching schematic diagram of a neural network optical flow estimation method implemented by FPGA according to an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the position of the smoothing operation matching block of the neural network optical flow estimation method implemented by FPGA provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The implementation principle of the method of the present invention is described in detail below in conjunction with the accompanying drawings:

[0047] Figure 1 The flowchart of the neural network optical flow estimation method implemented by FPGA provided by an embodiment of the present invention is shown, and the method includes:

[0048] Obtain two consecutive frames of pictures, and preprocess the pictures to obtain image blocks;

[0049] According to the image block, three layers of pyramid blocks are sampled;

[0050] Based on the three-layer pyramid blocks, binary neural network is used for feature extraction;

[0051] According to the result of feature extraction, a feature pyramid is obtained;

[0052] Output optical flow information based on the feature pyramid.

[0053] It should be noted that the image block includes a first image block obtained by preprocessing the first frame image and a second image block obtained by preprocessing the second frame image, and the three-layer pyramid block includes a first three-layer pyramid block sampled from the first image block and a second three-layer pyramid block sampled from the second image block.

[0054] Specifically, two consecutive frames of pictures are obtained, and preprocessed respectively to obtain a first image block of the first frame and a second image block of the second frame; then, according to the first image block of the first frame, a first three-layer pyramid block of the first frame is sampled, and according to the second image block of the second frame, a second three-layer pyramid block of the second frame is sampled; then, according to the first three-layer pyramid block of the first frame and the second three-layer pyramid block of the second frame, feature extraction is performed using a binary neural network respectively; then, according to the first three-layer pyramid block of the first frame and the second three-layer pyramid block of the second frame, feature extraction is performed to obtain a feature pyramid; finally, optical flow information is output according to the feature pyramid.

[0055] In some embodiments, obtaining two consecutive frames of pictures and performing preprocessing on the pictures to obtain image blocks includes:

[0056] In the same clock cycle, the pixels of the image are acquired one by one;

[0057] The pixels are delayed to obtain image blocks.

[0058] Specifically, according to the same clock cycle, the pixels of the image are acquired one by one, and then the pixels are delayed for a certain time to obtain an image block of a fixed size.

[0059] In some embodiments, according to the image block, a three-layer pyramid block is sampled, including:

[0060] Downsampling is performed according to the image block to obtain a downsampled image block;

[0061] Performing original resolution sampling according to the image block to obtain the original image block;

[0062] Upsampling is performed on the image block to obtain an upsampled image block.

[0063] In some embodiments, based on the three-layer pyramid block, a binary neural network is used to extract features, including:

[0064] Convert the three-layer pyramid blocks from parallel data stream to serial data stream;

[0065] According to the serial data stream, time-division feature extraction is performed using a time-division multiplexed binary neural network.

[0066] It should be noted that the three-layer pyramid block includes a first three-layer pyramid block obtained by sampling the first image block of the first frame image and a second three-layer pyramid block obtained by sampling the second image block of the second frame image. The first three-layer pyramid block and the second three-layer pyramid block are respectively converted from parallel data streams into serial data streams, and then two time-division multiplexed binary neural networks are used to perform feature extraction on the two serial data respectively.

[0067] In some embodiments, time-division feature extraction is performed using a time-division multiplexed binary neural network, including:

[0068] The feature extraction of the downsampled image block is reused once, the feature extraction of the original image block is reused once, and the feature extraction of the upsampled image block is reused four times.

[0069] Specifically, the time-division multiplexed binary neural network has three layers, and the network is reused six times in total, namely: feature extraction of the downsampled image block is reused once, feature extraction of the original image block is reused once, and feature extraction of the upsampled image block is reused four times.

[0070] In some embodiments, outputting optical flow information according to the feature pyramid includes:

[0071] Perform block matching operation according to the feature pyramid to obtain the first optical flow vector;

[0072] A smoothing operation is performed on the first optical flow vector to obtain a second optical flow vector.

[0073] Specifically, the feature pyramid is calculated hierarchically:

[0074] First, the first frame image features are divided into matching blocks of the same size;

[0075] Reference Figure 2 , for each matching block, block matching is performed within the search range w of the initial search point of the second frame image feature (the first layer defaults to the location of the matching block as the search point). The purpose of block matching is to find the block with the lowest matching cost with the matching block within the search range of the second frame image, and use the relative position of the two blocks as the optical flow vector of the matching block.

[0076] The matching cost of two pixels is the Hamming distance of their feature vectors, while the matching cost between blocks is the sum of the matching costs of the corresponding pixels in the blocks, which can be expressed as:

[0077] C b (B1, B2)=∑popcount(f1xor f2)

[0078] Where B1 and B2 are two image blocks for calculating the matching cost; f1 and f2 are feature vectors of pixels corresponding to the two image blocks; popcount() refers to the number of 1s in the statistical vector; and the xor symbol represents the bitwise exclusive OR operation on the two vectors.

[0079] After block matching, a rough optical flow vector of the layer is obtained, and then a smoothing operation is performed;

[0080] Reference Figure 3 , for each matching block, calculate the smoothing cost of its own optical flow vector and the optical flow vectors of the surrounding 8 matching blocks, and select the optical flow vector with the minimum cost to replace the optical flow vector of the original matching block. The smoothing cost calculation formula is as follows:

[0081]

[0082] Where B1 is the image block to be matched; Represents the selected optical flow vector; B j Represents the optical flow vector The matching block corresponding to the next frame image; λ is the smoothing coefficient; i represents the i-th matching block; is the optical flow vector of the 8 matching blocks around B1.

[0083] The obtained optical flow vector array is then upsampled and passed to the second layer, and the position pointed to by the optical flow vector of the matching block is used as the initial search point of the matching block. The upper layer operation is then repeated to perform block matching on each matching block within the initial search point range w. Then a smoothing operation is performed to obtain a new optical flow vector array. The array is upsampled and passed to the third layer.

[0084] In some embodiments, outputting optical flow information according to the feature pyramid further includes:

[0085] Performing a thinning operation on the second optical flow vector, and performing a median filter on the result of the thinning operation to obtain an optical flow map;

[0086] According to the preset period and optical flow map, the optical flow vector of each pixel in the optical flow map is output one by one.

[0087] Specifically, when processing the third layer image, a refinement operation is added to the initial search point range w search step:

[0088] In order to obtain the sub-pixel optical flow vector, the matching block with the minimum cost is compared with the eight surrounding matching blocks with a difference of one pixel. If the matching cost of the matching blocks is similar, the final optical flow vector can be taken as the weighted average of the optical flow vectors corresponding to the matching blocks. The specific calculation formula and calculation process are as follows:

[0089]

[0090] Where the C matrix is ​​the cost between the minimum cost matching block and the eight surrounding matching blocks that differ by one pixel.

[0091]

[0092]

[0093] Where K is the corresponding matching cost similarity matrix (k1 is 1 if the two matching blocks are similar, and k1 is 0 if the two matching blocks are not similar). i In the formula, t is the similarity threshold. When the ratio of the difference between a matching block cost and c0 to c0 is less than t, the matching block is considered to be similar to the matching block represented by c0.

[0094]

[0095]

[0096] Where P and Q are the weight matrices corresponding to the optical flow vector components u and v.

[0097]

[0098]

[0099] In the formula, u d It is the result of multiplying the corresponding elements of the K matrix and the P matrix and then dividing by 6. It represents the refined sub-pixel offset of the optical flow vector u. Similarly, v d It is the result of multiplying the corresponding elements of the K matrix and the Q matrix and then dividing by 6. It represents the refined sub-pixel offset in the direction of the optical flow vector v.

[0100]

[0101]

[0102] In the formula, Represents the optical flow vector result before refinement, where u0 and v0 are its components in two directions. That is the result after refinement, and its corresponding component is the original component plus the sub-pixel precision offset in its direction.

[0103] Finally, the final result of the third layer is subjected to median filtering to obtain an optical flow map, and the optical flow vectors of each pixel in the optical flow map are output one by one according to a preset period.

[0104] The embodiment of the present invention provides a neural network optical flow estimation device implemented by FPGA, comprising:

[0105] The first module is used to obtain two consecutive frames of pictures and pre-process the pictures to obtain image blocks;

[0106] The second module is used to sample and obtain three-layer pyramid blocks according to the image blocks;

[0107] The third module is used to extract features using a binary neural network based on the three-layer pyramid blocks;

[0108] The fourth module is used to obtain a feature pyramid based on the result of feature extraction;

[0109] The fifth module is used to output optical flow information based on the feature pyramid.

[0110] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0111] An embodiment of the present invention provides an electronic device, including a processor and a memory;

[0112] The memory is used to store programs;

[0113] The processor executes the program to implement the above-mentioned FPGA-implemented neural network optical flow estimation method.

[0114] The contents of the method embodiments of the present invention are all applicable to the electronic device embodiments. The functions specifically implemented by the electronic device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0115] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0116] In summary, the present invention aims at the problem that the optical flow calculation in the prior art is huge, a large number of computing units are consumed, or the calculation accuracy is low, and the practicality is not high. A neural network optical flow estimation method implemented by FPGA is proposed. By processing two consecutive frames of images, a three-layer pyramid image is first generated, which is a downsampled image, an original image, and an upsampled image. Subsequently, two time-division multiplexed binary neural networks are used to parallelly calculate the features corresponding to the pyramid image in the form of a stream, and then the feature pyramid is layered block matched to calculate the cost, and finally high-precision optical flow information is obtained through filtering, smoothing, and refinement operations, which simplifies the calculation amount and realizes high-precision optical flow estimation. In addition, the above method flow can conveniently realize end-to-end pipeline design, improve real-time performance, and pass simulation and synthesis, and can be actually deployed in an ALTERA FPGA to meet the real-time requirements of most real-time systems.

[0117] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0118] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0119] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0120] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0121] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0122] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0123] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0124] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0125] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A neural network optical flow estimation method implemented by FPGA, characterized in that: include: Obtain two consecutive frames of pictures, and perform preprocessing on the pictures to obtain image blocks; According to the image block, sampling is performed to obtain a three-layer pyramid block; The image block includes a first image block obtained by preprocessing a first frame image and a second image block obtained by preprocessing a second frame image, and the three-layer pyramid block includes a first three-layer pyramid block sampled from the first image block and a second three-layer pyramid block sampled from the second image block; and the three-layer pyramid block is sampled according to the image block, comprising: Downsampling the image block to obtain a downsampled image block; Performing original resolution sampling on the image block to obtain an original image block; Upsampling the image block to obtain an upsampled image block; According to the three-layer pyramid block, a binary neural network is used to extract features; The method of extracting features using a binary neural network based on the three-layer pyramid block includes: The first three-layer pyramid block and the second three-layer pyramid block are converted from parallel data streams into serial data streams, and then two time-division multiplexed binary neural networks are used to extract features from the two serial data respectively; According to the result of the feature extraction, a feature pyramid is obtained; Optical flow information is output according to the feature pyramid.

2. The neural network optical flow estimation method implemented by FPGA according to claim 1, characterized in that: The step of acquiring two consecutive frames of pictures and performing preprocessing on the pictures to obtain image blocks includes: In the same clock cycle, pixels of the image are acquired one by one; Delay processing is performed on the pixels to obtain the image block.

3. The neural network optical flow estimation method implemented by FPGA according to claim 1, characterized in that: The method of performing time-sharing feature extraction using the binary neural network with time-sharing multiplexing includes: The feature extraction of the down-sampled image block is multiplexed once, the feature extraction of the original image block is multiplexed once, and the feature extraction of the up-sampled image block is multiplexed four times.

4. The neural network optical flow estimation method implemented by FPGA according to claim 1, characterized in that: The outputting optical flow information according to the feature pyramid includes: Perform a block matching operation according to the feature pyramid to obtain a first optical flow vector; A smoothing operation is performed on the first optical flow vector to obtain a second optical flow vector.

5. The FPGA-implemented neural network optical flow estimation method according to claim 4, characterized in that: Outputting optical flow information according to the feature pyramid also includes: Performing a thinning operation on the second optical flow vector, and performing median filtering on a result of the thinning operation to obtain an optical flow map; According to a preset period and the optical flow map, the optical flow vector of each pixel in the optical flow map is output one by one.

6. A neural network optical flow estimation device implemented by FPGA, characterized in that: include: The first module is used to obtain two consecutive frames of pictures and perform preprocessing on the pictures to obtain image blocks; The second module is used to sample and obtain a three-layer pyramid block according to the image block; The image block includes a first image block obtained by preprocessing a first frame image and a second image block obtained by preprocessing a second frame image, and the three-layer pyramid block includes a first three-layer pyramid block sampled from the first image block and a second three-layer pyramid block sampled from the second image block; and the three-layer pyramid block is sampled according to the image block, comprising: Downsampling the image block to obtain a downsampled image block; Performing original resolution sampling on the image block to obtain an original image block; Upsampling the image block to obtain an upsampled image block; A third module is used to extract features using a binary neural network according to the three-layer pyramid block; The method of extracting features using a binary neural network based on the three-layer pyramid block includes: The first three-layer pyramid block and the second three-layer pyramid block are converted from parallel data streams into serial data streams, and then two time-division multiplexed binary neural networks are used to extract features from the two serial data respectively; The fourth module is used to obtain a feature pyramid according to the result of the feature extraction; The fifth module is used to output optical flow information according to the feature pyramid.

7. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 5.

Images