Image processing apparatus and motion estimation method thereof

Abstract

An image processing apparatus for performing motion estimation is provided. The image processing apparatus includes at least one processor configured to implement: a position estimation module configured to estimate an initial search position by providing a first frame and a second frame of a video as inputs to a neural network trained to output the initial search position; and a motion estimation module configured to perform motion estimation based on the initial search position.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2023-0072735 filed on June 7, 2023 in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to an image processing device based on a neural network and a method for optimizing motion estimation of the image processing device. Background Art

[0004] With the development of information and communication technology, the ways of capturing, storing and sharing videos have become active and diverse. For example, video can be captured and stored using sports equipment and portable devices, and the captured video can be processed using image signal processing (ISP) to solve physical degradation, or codec technology can be used for efficient storage and transmission of video. In ISP or codec technology, video processing can be performed by estimating the correlation between frames in a sequence (e.g., an image stream) to improve video quality, or the correlation can be compressed so that low-volume videos can be stored and transmitted. The correlation between frames can be determined based on motion estimation (ME) between images in a unit (e.g., a small block or block) of the video to be processed. However, if a maximum search range is set within a system on chip (SoC) (e.g., a sports device), or if a search range for software optimization is set, motion exceeding a certain level may not be found (the level may increase as the video resolution increases). Summary of the invention

[0005] Provided are a neural network-based image processing device and a method for optimizing motion estimation of the image processing device.

[0006] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments.

[0007] According to one aspect of the present disclosure, an image processing device includes at least one processor, which is configured to implement: a position estimation module, configured to estimate an initial search position by providing a first frame and a second frame of a video as input to a neural network, the neural network being trained to output the initial search position; and a motion estimation module, configured to perform motion estimation based on the initial search position.

[0008] The neural network can be trained to output the initial search position as an affine matrix value.

[0009] The neural network may also include a convolutional neural network (CNN).

[0010] The first frame and the second frame may have original sizes, and the image processing device may also include a first size conversion module, which is configured to convert the first frame into a converted first frame having an input size associated with the neural network, and convert the second frame into a converted second frame having the input size.

[0011] The motion estimation module may also be configured to perform motion estimation by moving the search range toward an initial search position.

[0012] The image processing apparatus may further include a second size conversion module configured to convert the value of the initial search position into a conversion value having a target image size associated with the motion estimation module.

[0013] The image processing apparatus may further include a training module, which is differentiable and may be configured to train the neural network using a back propagation method.

[0014] The training module can also be configured to train the neural network using unsupervised learning.

[0015] The training module may also be configured to: train the neural network so as to minimize a loss function between the first training frame and the predicted image, wherein the predicted image is predicted based on the second training frame using the neural network.

[0016] The training module may also include: an affine transformation module, configured to receive the first training frame and the second training frame as input, and perform affine transformation on the second training frame based on the affine matrix value; a motion kernel estimation module, configured to perform motion estimation on the second training frame after the affine transformation, and output the motion kernel; and a motion compensation module, configured to perform motion compensation based on the result of the motion estimation, and output a predicted image.

[0017] The motion kernel estimation module can also be configured to: perform expansion for each block of the second training frame after affine transformation; divide each block into multiple small blocks; calculate the sum of absolute differences (SAD) between the multiple small blocks of each block and the corresponding block of the first training frame; and use a softmax function to generate a motion kernel based on the calculated SAD.

[0018] The image processing apparatus may further include at least one of a video codec device and an image signal processing (ISP) device.

[0019] According to one aspect of the present disclosure, a motion estimation method for performing motion estimation on a video by an image processing device includes: estimating an initial search position by providing a first frame and a second frame of the video as input to a neural network, the neural network being trained to output the initial search position; and performing motion estimation based on the initial search position.

[0020] The neural network can be trained to output the initial search position as an affine matrix value.

[0021] The first frame and the second frame may have original sizes, and the method may further include converting the first frame into a converted first frame having an input size associated with the neural network, and converting the second frame into a converted second frame having the input size.

[0022] Performing motion estimation may also include moving the search range toward an initial search position.

[0023] The method may further include converting the value of the initial search position into a conversion value having a target image size associated with the motion estimation.

[0024] The method may also include: training the neural network using a differentiable training module and a back-propagation method.

[0025] Training of neural networks can be performed using unsupervised learning.

[0026] According to one aspect of the present disclosure, an apparatus for optimizing motion estimation of a video in a video codec device or an image signal processing (ISP) device includes: one or more processors, wherein the one or more processors are configured to: read a current frame and a reference frame of a video from a memory before the video codec device or the ISP device performs motion estimation of the video, use a neural network to obtain an initial search position for motion estimation, and provide the initial search position to the video codec device or the ISP device, wherein the neural network is trained to receive a current frame and a reference frame as input, and output an affine matrix value representing the initial search position. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1A is a block diagram showing an image processing apparatus according to an embodiment;

[0028] Figure 1B is a diagram showing an example of a neural network based position estimator according to an embodiment;

[0029] Figure 2 is a block diagram showing an image processing apparatus according to another embodiment;

[0030] Figure 3 is a block diagram showing an image processing apparatus according to yet another embodiment;

[0031] Figure 4A , Figure 4B and Figure 4C According to the description of the embodiment Figure 3 Figure 1 shows an example of a trainer in ;

[0032] Figure 5 is a block diagram showing an apparatus for optimizing motion estimation of a video according to an embodiment;

[0033] Fig. 6A and Figure 6B is a diagram illustrating an example of use of a standard codec according to an embodiment;

[0034] Figure 7 is a flow chart showing a motion estimation method according to an embodiment;

[0035] Figure 8 is a flow chart illustrating a method of training a neural network according to an embodiment;

[0036] Fig. 9 is a diagram showing a Figure 8 Flowchart of an example of motion kernel estimation. DETAILED DESCRIPTION

[0037] Details of some embodiments are included in the following detailed description and drawings. According to the embodiments described in detail below with reference to the drawings, the advantages and features of the present disclosure and their implementation methods will be more clearly understood. Throughout the drawings and detailed description, unless otherwise described, the same reference numerals will be understood to refer to the same elements, features and structures.

[0038] It should be understood that although the terms "first", "second", etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish elements from each other. Unless otherwise explicitly stated, any reference to the singular may include the plural. In addition, unless explicitly described to the contrary, expressions such as "including" or "comprising" will be understood to mean including the elements described, but not excluding any other elements. In addition, terms such as "unit" or "module" should be understood as units that perform at least one function or operation and can be embodied as hardware, software or a combination thereof.

[0039] As common in the art, the embodiment is described and shown in the accompanying drawings in units of functional blocks, units and / or modules. It will be appreciated by those skilled in the art that these blocks, units and / or modules are physically implemented by electronic (or optical) circuits such as logic circuits, discrete components, microprocessors, hard-wired circuits, memory elements, wiring connections, etc., wherein the electronic (or optical) circuits can be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. In the case where blocks, units and / or modules are implemented by microprocessors, etc., they can be programmed using software (e.g., microcode) to perform the various functions discussed herein, and can be optionally driven by firmware and / or software. Alternatively, each block, unit and / or module can be implemented by dedicated hardware or implemented as a combination of dedicated hardware that performs some functions and a processor (e.g., one or more programmed microprocessors and associated circuits) that performs other functions. In addition, without departing from the scope of this article, each block, unit and / or module of the embodiment can be physically divided into two or more interactive and discrete blocks, units and / or modules. Furthermore, the blocks, units and / or modules of the embodiments may be physically combined into more complex blocks, units and / or modules without departing from the scope of this document.

[0040] Figure 1A is a block diagram showing an image processing apparatus according to an embodiment of the present disclosure. Figure 1B is a diagram showing an example of a neural network-based position estimator 110 according to an embodiment of the present disclosure.

[0041] refer to Figure 1A , the image processing apparatus 100 may include a position estimator 110 and a motion estimator 120. In an embodiment, the position estimator 110 may be referred to as a position estimation module, and the motion estimator 120 may be referred to as a motion estimation module.

[0042] The image processing apparatus 100 may be one or more of a video codec device for performing video encoding, or a video stabilizer, and an image signal processing (SIP) device for solving physical degradation including noise reduction (NR), high dynamic range (HDR), deblurring, frame rate up conversion (FRUC), etc. In this case, the position estimator 110 and the motion estimator 120 may be implemented as separate hardware or software modules, or the position estimator 110 may be integrated into the motion estimator 120 to be implemented as one module. In addition, the motion estimator 120 of the image processing apparatus 100 may be included as a component in a codec device or an ISP device, and the position estimator 110 may be implemented separately from the video codec device or the ISP device to serve as a preprocessor before these devices.

[0043] The position estimator 110 may receive a video captured by at least one of a camera and an inertial measurement unit, may estimate an initial search position for optimizing motion estimation of the video, and may provide information related to the estimated initial search position to the motion estimator 120 .

[0044] refer to Figure 1B , the position estimator 110 may be based on a neural network 111, or otherwise relate to, or include a neural network 111. The neural network 111 may output information related to an initial search position by using a first frame and a second frame as input. For example, at a specific time t, the first frame may be a current frame corresponding to time t, and the second frame may be a reference frame corresponding to time t-1. The neural network 111 may be pre-trained to output an affine matrix value ISP representing an initial search position. The neural network 111 may be, for example, a convolutional neural network (CNN), but the embodiment is not limited thereto. The neural network 111 may be trained using a back propagation method. In addition, the neural network 111 may be trained using unsupervised learning.

[0045] The motion estimator 120 may perform motion estimation (ME) by using the affine matrix value ISP representing the initial search position output by the position estimator 110, and the first frame and the second frame. The motion estimator 120 may perform motion estimation in units of blocks between the first frame and the second frame. For example, the motion estimator 120 may perform motion estimation by moving the center of the search range toward the initial search position.

[0046] Typically, codec encoding or motion estimation in image signal processing can be performed by using a full search method, in which a search range is specified based on the block position in the current frame, and the block is moved one pixel at a time within the search range of the reference frame to find an area with the minimum sum of absolute differences (SAD) between the pixels in the block, thereby obtaining a motion vector. The full search method uses a large amount of computing resources, however, using a simpler method may reduce the accuracy of motion vector estimation. In addition, if the search range is increased for high-resolution image processing, the amount of calculation may increase, resulting in performance degradation. If the motion estimator is configured as hardware, the search range is fixed, thereby limiting the search range. For example, if a maximum search range is set within a system on chip (SoC) (e.g., in a device such as a motion device), or if the search range is set for software optimization, as the resolution of the video increases, motion exceeding a certain level may not be found or detected, so a new hardware encoder or software architecture may be required.

[0047] According to an embodiment of the present disclosure, by using a neural network trained to output affine matrix values ​​to estimate the initial search position, the search range can be controlled in real time during motion estimation of a video, thereby increasing the accuracy of motion estimation, even with a relatively small amount of calculation. In addition, in this way, the compression rate and quality of the video can be improved.

[0048] Figure 2 is a block diagram showing an image processing apparatus according to another embodiment of the present disclosure.

[0049] refer to Figure 2 , the image processing apparatus 200 may include a position estimator 110, a motion estimator 120, a first size converter 210, and a second size converter 220. In an embodiment, the first size converter 210 may be referred to as a first size conversion module, and the second size converter 220 may be referred to as a second size conversion module.

[0050] The first size converter 210 may convert a first size of an input raw image into a second size corresponding to an input of the position estimator 110. For example, a video captured by a camera may have various resolutions such as full high definition (FHD), 4K, 8K, etc. In contrast, the neural network of the position estimator 110 may be trained to process an image of a predetermined size (e.g., 448×256) with a relatively small amount of calculation.

[0051] The first size converter 210 may convert an input image having a first size into a converted image having a set or predetermined second size associated with or suitable for the neural network by enlarging or reducing the input image having the first size, and may provide the converted image having the second size to the position estimator 110.

[0052] The position estimator 110 may output an affine matrix value representing an initial search position by using the converted first frame and second frame having the second size as inputs.

[0053] The second size converter 220 may scale up or down the affine matrix value ISP output by the position estimator 110 to a scaling value having a target image size associated with or suitable for the motion estimator 120 , and may provide the scaling value to the motion estimator 120 .

[0054] The motion estimator 120 may perform motion estimation by moving the search range based on the affine matrix value ISP representing the initial search position. In this way, it is possible to effectively respond to inputs of images having various sizes.

[0055] Figure 3 is a block diagram showing an image processing apparatus according to still another embodiment of the present disclosure. Figure 4A , Figure 4B and Figure 4C Yes Description Figure 3 Figure 2. An example of a trainer in .

[0056] refer to Figure 3 , the image processing apparatus 300 may include a position estimator 110, a motion estimator 120, and a trainer 310. In an embodiment, the trainer 310 may be referred to as a training module. The position estimator 110 and the motion estimator 120 may be the same as those described above. Figure 1A to Figure 1B and Figure 2 The described position estimator 110 and motion estimator 120 are substantially similar, and thus redundant or repeated descriptions may be omitted.

[0057] The trainer 310 may be implemented by or using a neural network that simulates a standard codec or ISP. The trainer 310 may include a differentiable module so that back propagation flows when training the neural network 111.

[0058] For example, refer to Figure 4A , the trainer 310 may include a differentiable affine transformer 410, a motion kernel estimator 420, and a motion compensator 430. In an embodiment, the affine transformer 410 may be referred to as an affine transformation module, the motion kernel estimator 420 may be referred to as a motion kernel estimation module, and the motion compensator 430 may be referred to as a motion compensation module.

[0059] If an affine matrix value ISP representing an initial search position is output by inputting a first frame FR1 for training (hereinafter referred to as the first training frame FR1) and a second frame FR21 for training (hereinafter referred to as the second training frame) into the neural network 111, the affine transformer 410 can perform an affine transformation on the second training frame FR21 based on the affine matrix value ISP. In this way, the second training frame FR21 can be converted into an image similar to the first training frame FR1, so that motion estimation for each block can be performed within the search range.

[0060] Then, the motion kernel estimator 420 can divide each of the first training frame FR1 and the second training frame FR22 after affine transformation into a plurality of (N) blocks BL1 and BL2 of predetermined sizes, and can perform motion estimation for each block. In the first training frame FR1, each block can be divided into an image of a fixed size, and in the second training frame FR22, each block can be divided into a size that covers the search range to overlap with surrounding blocks. The motion kernel estimator 420 can output a motion in a kernel form, which can be used for subsequent differentiable convolution operations during prediction using motion compensation.

[0061] Figure 4Bis a diagram showing an example of the motion kernel estimator 420 which is set to be differentiable, and Figure 4C is a diagram showing an example of processing performed by the expansion module 421.

[0062] refer to Figure 4B and Figure 4C Each block BL2 of the second training frame FR22 after affine transformation can be divided into (2s+1) by the expansion module 421. 2 Here, S may represent a search range. The size of the small block PA may be a block size B, and may be equal to the block size of the first training frame FR1. For example, if the block size B is 4, and the search range S is 3, the block may be divided into 49 small blocks PA of size 4 by the expansion module 421.

[0063] Then, the SAD calculation module 422 can calculate the SAD between the small blocks PA of the specific block BL2 of the second training frame FR22 after affine transformation and the block BL1 corresponding to the specific block BL2 of the first training frame FR1, and the motion kernel MK can be generated by selecting the small block position with the smallest SAD using the softmax function 423.

[0064] The motion compensator 430 may perform motion compensation and / or convolution by using the motion kernel MK output by the motion kernel estimator 420, and may generate a prediction image FR3 for the first training frame FR1 by merging the motion compensated blocks at their original positions. The motion compensator 430 may output the generated prediction image FR3, or a residual image which may be a difference FR1-FR3 between the first training frame FR1 and the prediction image FR3.

[0065] The trainer 310 may set various loss functions and may train the neural network 111 so that the residual between the predicted image FR3 and the first training image FR1 may be minimized. For example, the loss function may include a peak signal-to-noise ratio (PSNR), a mean square error (MSE), a cross entropy loss, a binary cross entropy loss, a log-likelihood loss, a frequency domain loss, etc., but the embodiment is not limited thereto. As described above, in an embodiment, the neural network 111 may be trained by unsupervised learning using the predicted image FR3 without obtaining accurate real data (GT).

[0066] Figure 5 is a block diagram of an apparatus for optimizing motion estimation of a video according to an embodiment of the present disclosure.

[0067] refer to Figure 5 The apparatus 500 may be an apparatus for optimizing video motion estimation in a video coding and decoding device or an image signal processing device, and may include one or more processors 510 .

[0068] Before performing motion estimation of a video through the video codec device 531 or the ISP device 532, the processor 510 may obtain an initial search position for motion estimation by using a neural network by reading a current frame and a reference frame of a video captured by at least one of a camera and an IMU from the memory 520. In addition, the processor 510 may provide information related to the obtained initial search position to the codec device 531 or the ISP device 532. In an embodiment, the reference frame may be a frame before the current frame.

[0069] For example, the neural network may be a CNN. The neural network may be pre-trained by unsupervised learning and a back-propagation method using a trainer implemented as a differentiable module simulating a codec device or an ISP device. The neural network may be pre-trained to output an affine matrix value representing an initial search position in a search range.

[0070] The processor 510 can convert the size of the frame read from the memory 520 to fit the input of the neural network, and then can input the frame to the next neural network, can convert the affine matrix value output by the neural network to a size suitable for the target image output by the video codec device 531 or the ISP device 532, and can provide the value to at least one of the video codec device 531 and the ISP device 532.

[0071] Fig. 6A and Figure 6B A diagram illustrating a usage example in a standard codec.

[0072] Fig. 6A is a diagram showing an example of applying an affine transform during inter-frame prediction in High Efficiency Video Coding (HEVC) (which may also be referred to as H.265). Fig. 6A, a coding unit (CU) 611 of a current frame 610 at time t and a CU 621 of a reference frame 620 at time t-1 are shown, and the frame is divided into 16 blocks. In HEVC, a CU may be referred to as a prediction unit (PU). The processing of CU 611 of the current frame 610 may include searching for matching motion in a reference frame 620, in which case the center of the search range may be moved based on the initial search position by estimating an initial search position using a pre-trained neural network. Typically, affine transformations including rotations of CUs or PUs may not be feasible in HEVC, so that more similar geometries may be found by moving the search range. In addition, in the case of rolling rotation, blocks with different angles may not be found, making it impossible to find an exact matching block. However, according to an embodiment of the present disclosure, by finding an area with the minimum SAD within the affine transformation area, the residual component can be reduced, thereby improving the compression rate, compared with the case of a limited search range.

[0073] Figure 6B is a diagram showing an example of applying an affine transform during inter-frame prediction in Versatile Video Coding (VVC) (which may also be referred to as H.266). In VVC, affine transforms of PUs may be supported, and the encoder consumes a relatively large amount of resources. Figure 6B , PU 631 of the current frame 630 is shown on the left, and PU 641 of frame 640 to which affine transformation is applied is shown in the middle. In an embodiment, unlike the general method of finding motion for each PU, affine motion can be found after applying affine bias, so that the accuracy of the motion vector relative to the search range can be improved. Then, by performing motion compensation in units of 4×4 sub-blocks of PU 641, only affine information can be sent to the decoder. As described above, by optimizing motion estimation using a neural network, the amount of affine operations of the encoder can be reduced.

[0074] Because the ISP may not have a setting standard such as a standard codec, according to an embodiment of the present disclosure, the affine information output from the neural network can be easily used. For example, by using the affine information for the video stabilizer, the inter-frame motion during video shooting can be estimated and compensated in real time based on the above embodiment, so that the camera movement can be offset. For noise reduction, there is a method for reducing random image noise (e.g., 3DNR, etc.) by overlapping similar noise patches, wherein the motion can be used in the process of estimating the patches based on the surrounding frames. In this case, the patches can be estimated by specifying a search range as in motion estimation, and the patches can be obtained by the affine transformation according to the above embodiment, more similar geometric structures can be used, and the noise removal performance can be improved. In HDR technology, brightness adaptive fusion between two images with different exposures can be performed, in which case the affine information according to the above embodiment can be used as the camera motion affine information between the two images with a time difference. In addition, the above affine transformation can be used to replace the camera motion during the exposure time in the camera motion deblurring algorithm, and the deblurring kernel can be estimated and reflected. In addition, the above embodiment can be used for inter-frame motion estimation during frame rate up-conversion.

[0075] Figure 7 is a flowchart illustrating a motion estimation method according to an embodiment of the present invention.

[0076] Figure 7 The method is an example of a motion estimation method that can be performed by an image processing device (for example, at least one of the above-mentioned image processing devices 100, 200 and 300, and device 500).

[0077] refer to Figure 7 , at operation 710, the image processing apparatus may estimate an initial search position by inputting two frames of the video to a neural network. In this case, the neural network may be trained by unsupervised learning using a back-propagation method and a trainer that may be implemented as a differentiable module. The neural network may be trained to output an initial search range in the form of an affine matrix value, which may be used for motion estimation in codec encoding and ISP.

[0078] At operation 720, the image processing apparatus may estimate the motion of the video based on information related to the initial search position output by the neural network. For example, by estimating the motion after moving the center of the search range to the initial search position, the amount of calculation may be minimized.

[0079] Figure 8 is a flowchart illustrating a method of training a neural network according to an embodiment of the present disclosure. Fig. 9 It shows Figure 8Flowchart of an example of motion kernel estimation. In an embodiment, Figure 8 and Fig. 9 One or more operations shown in the figure may be performed by or using any one of the above-mentioned elements (eg, the image processing apparatus 300 and at least one of the elements included therein).

[0080] refer to Figure 8 , at operation 810, an affine matrix value representing an initial search position may be obtained by inputting the first training frame and the second training frame into a neural network.

[0081] Then, based on the affine matrix value output at operation 810, at operation 820, an affine transformation can be performed on the second training frame by using an affine transformation module of the trainer. The trainer can be implemented by or using a neural network that simulates a standard codec or ISP. The affine transformation model can be implemented as a differentiable module so that back propagation flows when training the neural network. In this way, the second training frame FR21 can be converted into an image similar to the first training frame FR1, so that motion estimation for each block can be performed within the search range.

[0082] Subsequently, based on the affine-transformed second training frame, a motion kernel may be estimated at operation 830 .

[0083] refer to Fig. 9 At operation 910, each block of the second training frame after affine transformation may be divided into a plurality of small blocks by performing expansion. In this case, each block of the second training frame may be divided into a size that covers the search range to overlap with surrounding blocks.

[0084] Then, at operation 920, SADs between the small blocks of each block of the second training frame after affine transformation and the blocks of the first training frame corresponding to each block may be calculated.

[0085] Subsequently, by applying a softmax function to the SAD calculated for each block of the second training frame after affine transformation at operation 930 , a motion kernel may be output by selecting a small block position having a minimum SAD at operation 940 .

[0086] Reference again Figure 8 , motion compensation and / or convolution may be performed by using the motion kernel estimated at operation 830, and a prediction image for the first training frame may be generated at operation 840 by merging the motion compensated blocks at their original positions. By using the generated prediction image FR3, or a residual image which may be a difference FR1-FR3 between the first training frame FR1 and the prediction image FR3, the neural network 111 may be trained to minimize a predefined loss function.

[0087] The embodiments may be implemented as computer-readable codes written on a computer-readable recording medium.The computer-readable recording medium may be any type of recording device that stores data in a computer-readable manner.

[0088] Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and carrier wave (e.g., data transmission via the Internet). Computer-readable recording media can be distributed on multiple computer systems connected to the network so that computer-readable codes are written therein and executed therefrom in a decentralized manner. Those of ordinary skill in the art to which the present disclosure belongs can easily infer the functional programs, codes, and code segments required to implement the embodiments.

[0089] Various embodiments of the present disclosure have been described herein. However, it will be apparent to those skilled in the art that various changes and modifications may be made without changing the scope of the present disclosure. Therefore, it is clear that the above embodiments are illustrative in all aspects and are not intended to limit the present disclosure.

Claims

1. An image processing device, include: At least one processor configured to implement: a position estimation module configured to estimate an initial search position by providing a first frame and a second frame of the video as inputs to a neural network, the neural network being trained to output the initial search position; as well as The motion estimation module is configured to perform motion estimation based on the initial search position.

2. The image processing device according to claim 1, in, The neural network is trained to output the initial search position as affine matrix values.

3. The image processing device according to claim 1, in, The neural network includes a convolutional neural network (CNN).

4. The image processing device according to claim 1, in, The first frame and the second frame have original sizes, and The image processing device further includes a first size conversion module, which is configured to convert the first frame into a converted first frame having an input size associated with the neural network, and convert the second frame into a converted second frame having the input size.

5. The image processing device according to claim 1, in, The motion estimation module is further configured to perform the motion estimation by moving a search range toward the initial search position. 6 . The image processing apparatus according to claim 1 , further comprising a second size conversion module configured to convert the value of the initial search position into a conversion value having a target image size associated with the motion estimation module. 7 . The image processing apparatus according to claim 2 , further comprising a training module, which is differentiable and configured to train the neural network using a back propagation method.

8. The image processing device according to claim 7, in, The training module is further configured to train the neural network using unsupervised learning.

9. The image processing device according to claim 7, in, The training module is further configured to train the neural network to minimize a loss function between a first training frame and a predicted image predicted based on a second training frame using the neural network.

10. The image processing device according to claim 9, in, The training module includes: an affine transformation module configured to receive the first training frame and the second training frame as input, and perform an affine transformation on the second training frame based on the affine matrix value; a motion kernel estimation module configured to perform the motion estimation on the second training frame after affine transformation and output a motion kernel; and The motion compensation module is configured to perform motion compensation based on the result of the motion estimation and output the predicted image.

11. The image processing device according to claim 10, in, The motion kernel estimation module is further configured to: Performing expansion on each block of the second training frame after the affine transformation; Dividing each of the blocks into a plurality of small blocks; Calculating absolute differences and SADs between the plurality of small blocks of each block and corresponding blocks of the first training frame; as well as The motion kernel is generated based on the calculated SAD using a softmax function.

12. The image processing device according to claim 1, in, The image processing apparatus includes at least one of a video coding and decoding device and an image signal processing (ISP) device.

13. A motion estimation method for performing motion estimation on a video by an image processing device, the motion estimation method include: estimating an initial search position by providing a first frame and a second frame of the video as inputs to a neural network, the neural network being trained to output the initial search position; as well as The motion estimation is performed based on the initial search position.

14. The motion estimation method according to claim 13, in, The neural network is trained to output the initial search position as affine matrix values.

15. The motion estimation method according to claim 13, in, The first frame and the second frame have original sizes, and The motion estimation method further includes converting the first frame into a converted first frame having an input size associated with the neural network, and converting the second frame into a converted second frame having the input size.

16. The motion estimation method according to claim 13, in, Performing the motion estimation includes moving a search range toward the initial search position.

17. The motion estimation method according to claim 13, further comprising: include: The value of the initial search position is converted to a converted value having a target image size associated with the motion estimation.

18. The motion estimation method according to claim 13, further comprising: include: The neural network is trained using a differentiable training module and a back-propagation method.

19. The motion estimation method according to claim 18, in, The training of the neural network is performed using unsupervised learning.

20. A device for optimizing video motion estimation in a video coding and decoding device or an image signal processing (ISP) device, the device include: One or more processors configured to: Before the motion estimation of the video is performed by the video codec device or the ISP device, a current frame and a reference frame of the video are read from a memory, using a neural network to obtain an initial search position for said motion estimation, and providing the initial search position to the video codec device or the ISP device, Wherein, the neural network is trained to receive the current frame and the reference frame as input, and output an affine matrix value representing the initial search position.

Images