Image processing method and apparatus

Abstract

The embodiment of the application discloses an image processing method and device, and relates to the field of images. The specific scheme is as follows: firstly, a current image frame and a reference image frame are acquired, the current image frame is sequentially down-sampled and up-sampled to obtain a processed current image frame, and the reference image frame is sequentially down-sampled and up-sampled to obtain a processed reference image frame. Then, the processed current image frame and the processed reference image frame are respectively divided into a plurality of current sub-image blocks and a plurality of reference sub-image blocks according to a preset division mode. A reference sub-image block with the minimum similarity to each current sub-image block is determined as a matching block of the current sub-image block in the plurality of reference sub-image blocks. A motion vector corresponding to each current sub-image block is obtained based on each current sub-image block and the matching block corresponding to the current sub-image block. Finally, the current image frame is encoded based on the motion vector.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image, and particularly relates to an image processing method and device. BACKGROUND

[0002] At present, when encoding an image frame, in order to reduce the amount of encoded data of the image frame, inter-frame encoding can be performed on the current image frame. When performing inter-frame encoding on the current image frame, a coded image block most similar to the current image block can be searched in a reference image frame as a matching block of the current image block, an offset between the current image block and the matching block is taken as a motion vector, and the motion vector is encoded, so that the decoding end can learn the position of the current image block according to the motion vector. SUMMARY

[0003] The embodiments of the present application provide an image processing method and device, which can more accurately calculate a motion vector and improve image quality.

[0004] In one aspect, an image processing method is provided, which includes: first, obtaining a current image frame and a reference image frame, performing down-sampling and up-sampling on the current image frame in sequence to obtain a processed current image frame, and performing down-sampling and up-sampling on the reference image frame in sequence to obtain a processed reference image frame. Then, dividing the processed current image frame and the processed reference image frame into a plurality of current sub-image blocks and a plurality of reference sub-image blocks according to a preset division manner respectively, determining a reference sub-image block with the smallest similarity to each current sub-image block in the plurality of reference sub-image blocks as a matching block of the current sub-image block. Based on each current sub-image block and the matching block corresponding to the current sub-image block, a motion vector corresponding to the current sub-image block is obtained. Finally, encoding the current image frame based on the motion vector.

[0005] Based on the present solution, by performing down-sampling and up-sampling on the current image frame and the reference image frame in sequence, the high-frequency part in the current image frame and the reference image frame which does not represent the main information can be reduced, and noise signals can be filtered. Then, by dividing the processed current image frame and the processed reference image frame into image blocks, the best matching block of each current sub-image block can be accurately obtained. Therefore, the motion vector obtained based on the best matching block is more accurate, and when the current image frame is encoded based on the motion vector, the size of the bit stream can be reduced, and the image quality can be improved.

[0006] In some embodiments, the current image frame is an original current image frame or an image frame obtained by scaling the original current image frame at least once by using a scaling factor K, and the reference image frame is an original reference image frame or an image frame obtained by scaling the original reference image frame at least once by using the scaling factor K.

[0007] Based on the scheme, the current image frame and the reference image frame can be scaled at least once to obtain current image frames and reference image frames of different sizes, and the current image frames and the reference image frames of the corresponding sizes are down-sampled and up-sampled to reduce the high-frequency part in the current image frame and the reference image frame which does not represent the main information, filter the noise signal, and then obtain the best matching block of each current sub-image block of each size according to the processed current image frame and the reference image frame.

[0008] In some embodiments, the distances between the plurality of reference sub-image blocks and the current sub-image block in the search range corresponding to the current sub-image block are calculated respectively. The distances between the plurality of reference sub-image blocks and the current sub-image block in the search range are regularized to obtain a plurality of candidate similarities. The reference sub-image block corresponding to the smallest similarity in the plurality of candidate similarities is determined as the matching block of the current sub-image block.

[0009] Based on the scheme, the distances between the plurality of reference sub-image blocks and the current sub-image block in the search range corresponding to the current sub-image block are calculated respectively. The distances between the plurality of reference sub-image blocks and the current sub-image block in the search range are regularized to obtain a plurality of candidate similarities. The reference sub-image block corresponding to the smallest similarity in the plurality of candidate similarities is determined as the matching block of the current sub-image block.

[0010] In some embodiments, the smallest similarity in the plurality of candidate similarities corresponding to each current sub-image block is the similarity corresponding to the current sub-image block.

[0011] In some embodiments, the method further comprises: determining a target motion vector from a first motion vector corresponding to a first current sub-image block and a second motion vector corresponding to a second current sub-image block according to a first similarity corresponding to the first current sub-image block and a second similarity corresponding to the second current sub-image block. The first current sub-image block is the image block after scaling the second current sub-image block.

[0012] In some embodiments, the first similarity and the second similarity are compared, and in the case that the first similarity is less than or equal to the second similarity, the first motion vector is determined as the target motion vector. In the case that the first similarity is greater than the second similarity, the second motion vector is determined as the target motion vector.

[0013] According to the scheme, since a group of motion vectors and similarities can be obtained at each size when the current image frame and the reference image frame are scaled, the similarities of the current image block at different sizes need to be compared and the best motion vector is determined. Moreover, when the best motion vector is determined, if the similarity of the large block is higher than that of the small block, the motion vector corresponding to the large block is taken as the best motion vector, and if the similarity of the large block is lower than that of the small block, the motion vector corresponding to the small block is taken as the best motion vector, so that the best motion vector can be selected at different sizes, and the accuracy of the motion vector is further improved. In addition, the current image frame can be divided into image blocks of different sizes for encoding according to the scheme. It can be understood that the smaller the similarity value is, the higher the similarity is, and the larger the similarity value is, the lower the similarity is.

[0014] In some embodiments, the encoding the current image frame based on the motion vector comprises encoding the current image frame based on the target motion vector.

[0015] In another aspect, an image processing apparatus is provided. The image processing apparatus comprises:

[0016] an obtaining module configured to obtain a current image frame and a reference image frame;

[0017] a sampling module configured to sequentially down-sample and up-sample the current image frame to obtain a processed current image frame, and sequentially down-sample and up-sample the reference image frame to obtain a processed reference image frame;

[0018] a dividing module configured to divide the processed current image frame and the processed reference image frame into a plurality of current sub-image blocks and a plurality of reference sub-image blocks according to a preset division manner;

[0019] a determining module configured to determine, among the plurality of reference sub-image blocks, a reference sub-image block with a minimum similarity to each current sub-image block as a matching block of the current sub-image block;

[0020] the determining module is further configured to obtain, based on each current sub-image block and the matching block corresponding to the current sub-image block, a motion vector corresponding to the current sub-image block;

[0021] an encoding module configured to encode the current image frame based on the motion vector.

[0022] In some embodiments, the current image frame is an original current image frame or an image frame obtained by scaling the original current image frame at least once using a scaling factor K, and the reference image frame is an original reference image frame or an image frame obtained by scaling the original reference image frame at least once using the scaling factor K.

[0023] In some embodiments, the determining module is specifically configured to: calculate distances between the current sub-image block and a plurality of reference sub-image blocks in a search range corresponding to the current sub-image block, respectively; perform regularization processing on the distances between the current sub-image block and the plurality of reference sub-image blocks in the search range to obtain a plurality of candidate similarities; and determine a reference sub-image block corresponding to a smallest similarity in the plurality of candidate similarities as a matching block of the current sub-image block.

[0024] In some embodiments, the smallest similarity in the plurality of candidate similarities corresponding to each current sub-image block is the similarity corresponding to the current sub-image block.

[0025] In some embodiments, the determining module is further configured to determine a target motion vector from a first motion vector corresponding to a first current sub-image block and a second motion vector corresponding to a second current sub-image block according to a first similarity corresponding to the first current sub-image block and a second similarity corresponding to the second current sub-image block, wherein the first current sub-image block is an image block obtained by scaling the second current sub-image block.

[0026] In some embodiments, the determining module is specifically configured to compare the first similarity and the second similarity, and determine the first motion vector as the target motion vector in a case where the first similarity is smaller than or equal to the second similarity, or determine the second motion vector as the target motion vector in a case where the first similarity is greater than the second similarity.

[0027] In some embodiments, the encoding module is specifically configured to encode the current image frame based on the target motion vector.

[0028] In another aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores computer program instructions, which, when executed on a computer (e.g., an image processing apparatus), cause the computer to perform the image processing method according to any one of the above embodiments.

[0029] In yet another aspect, a computer program product is provided. The computer program product includes computer program instructions, which, when executed on a computer (e.g., an image processing apparatus), cause the computer to perform the image processing method according to any one of the above embodiments.

[0030] In yet another aspect, a computer program is provided. When the computer program is executed on a computer (e.g., an image processing apparatus), the computer program causes the computer to perform the image processing method according to any one of the above embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the present disclosure, the following will briefly introduce the drawings needed to be used in some embodiments of the present disclosure. Obviously, the drawings described in the following description are only some drawings of the embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art according to these drawings. In addition, the drawings in the following description can be regarded as schematic diagrams, and are not limited to the actual size, actual process, actual timing of signals, etc. of the products involved in the embodiments of the present disclosure.

[0032] Figure 1 A structural diagram of an image encoder provided for an embodiment of the present application;

[0033] Figure 2 A flowchart of an image processing method provided for an embodiment of the present application;

[0034] Figure 3 An application diagram of an image processing method provided for an embodiment of the present application;

[0035] Figure 4 An application diagram of another image processing method provided for an embodiment of the present application;

[0036] Figure 5 An application diagram of still another image processing method provided for an embodiment of the present application;

[0037] Figure 6 An application diagram of yet another image processing method provided for an embodiment of the present application;

[0038] Figure 7 An application diagram of still another image processing method provided for an embodiment of the present application;

[0039] Figure 8 An application diagram of yet another image processing method provided for an embodiment of the present application;

[0040] Figure 9 An application diagram of still another image processing method provided for an embodiment of the present application;

[0041] Figure 10 An application diagram of yet another image processing method provided for an embodiment of the present application;

[0042] Figure 11 An application diagram of still another image processing method provided for an embodiment of the present application;

[0043] Figure 12 A flowchart of another image processing method provided for an embodiment of the present application;

[0044] Figure 13 An application diagram of still another image processing method provided for an embodiment of the present application;

[0045] Figure 14 FIG. 1 is a structural diagram of an image processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] The technical solutions in some embodiments of the present disclosure will be clearly and completely described below with reference to the drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments provided in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present disclosure.

[0047] Unless otherwise required by context, the term “comprise” and other forms of the term “comprise”, such as “comprises” and “comprising”, are used in an open, inclusive sense, that is, as “including, but not limited to”. In the description of the specification, the terms “one embodiment”, “some embodiments”, “exemplary embodiments”, “example”, “specific example” or “some examples” are intended to mean that a particular feature, structure, material, or characteristic included in at least one embodiment or example of the present disclosure. The illustrative representation of the above terms does not necessarily mean the same embodiment or example. In addition, the specific features, structures, materials or characteristics described can be included in any one or more embodiments or examples in any appropriate manner.

[0048] Hereinafter, the terms “first” and “second” are used only for descriptive purposes, and cannot be understood to indicate or imply relative importance or implicitly indicate the number of indicated technical features. Therefore, the features defined with “first” and “second” can explicitly or implicitly include one or more features. In the description of the embodiments of the present disclosure, unless otherwise stated, the meaning of “a plurality of” is two or more.

[0049] “A and / or B” includes the following three combinations: only A, only B, and a combination of A and B.

[0050] As used herein, the term "if' is, optionally, interpreted as meaning "when" or "upon" or "in response to a determination" or "in response to a detection of, depending on context. Similarly, the phrase "if determined," or "if [stated condition or event] is detected," is, optionally, interpreted as meaning "upon a determination" or "in response to a determination" or "upon a detection of [stated condition or event]" or "in response to a detection of [stated condition or event]," depending on context.

[0051] Additionally, use of "based on" means open and inclusive, in that a process, step, calculation, or other action "based on" one or more stated conditions or values can be based, in practice, on additional conditions or values beyond those stated.

[0052] Firstly, the nouns in the embodiments of the present disclosure are explained.

[0053] Down-sampling (or called down-sampling) refers to reducing the image frame, through which the image frame can be made to conform to the size of the display area, or a thumbnail of the image frame is generated. For example, a c-fold down-sampling is performed on an image frame with a size of M*N pixels, and an image frame with a size of (M / c)*(N / c) pixels is obtained, where c is a common divisor of M and N.

[0054] Up-sampling (or called image interpolation) refers to enlarging the image frame, through which the image frame can be displayed on a display device with higher resolution. Up-sampling almost always adopts an interpolation method, that is, on the basis of the original image frame pixels, a new element is inserted between the pixel points by using a suitable interpolation algorithm.

[0055] Intra-frame coding refers to a coding mode in which, when image compression processing is performed, discrete cosine transform (DCT), zigzag scanning, quantization processing, and variable length coding (VLC) are performed on the image frame.

[0056] Inter-frame coding refers to a coding mode in which, when image compression processing is performed, motion vectors and texture (prediction residual) information between image frames are coded by using the temporal redundancy of video images.

[0057] Next, the system architecture involved in the present disclosure is introduced.

[0058] Figure 1 is a structural diagram of an image encoder to which the method provided by the embodiments of the present disclosure is applied. The image encoder includes a residual calculation unit, a selection switch, a DCT module, a quantizer, a VLC module, a buffer, a rate control module, a dequantizer, an inverse DCT module, a frame storage module, and a motion estimation and compensation module.

[0059] As shown in Figure 1 The image encoder receives an image frame through an input interface. For example, the image frame can be one of a sequence of pictures forming a video or a video sequence. The image frame received by the image encoder can also be referred to as a current image frame or a to-be-encoded image frame. The image encoder can perform intra-frame encoding or inter-frame encoding on the image frame it receives.

[0060] The selection switch is used to select an intra-frame encoding image compression mode or an inter-frame encoding image compression mode when compressing an image. In order to better compress an image, intra-frame encoding is generally used for still images (pictures), and inter-frame encoding is generally used for moving images (videos).

[0061] The residual calculation unit is used to calculate a residual based on a current image frame and a predicted image frame. The DCT module is used to transform a spatial domain image to a frequency domain for image compression. In the spatial domain, image contents vary greatly, but in the frequency domain, after statistical analysis of a large number of images, it is found that the main components of the frequency coefficients of the images after DCT transformation are concentrated in a relatively small range and mainly located in the low frequency part. According to the statistical characteristics of the image signal in the frequency domain, some measures can be taken to discard the part with less energy in the frequency spectrum and try to retain the main frequency components in the transmission frequency spectrum, so as to achieve the purpose of image data compression.

[0062] The quantizer is used to further compress the data quantity by processing the frequency data processed by the DCT module again. Since the sensitivity of the human eye to various frequencies is different, the frequency data processed by the DCT module can be quantized. After quantization, the DCT coefficient matrix will have many zero values. Generally, the quotient of the data in the upper left corner is non-zero, and the quotient of the data in the lower right corner is very small, which can be rounded to 0 after rounding. Many 0 values appear on the coefficient matrix, greatly reducing the data quantity, on the one hand, retaining the main part of the image information, and on the other hand, compressing the image data.

[0063] The VLC module is used to encode the coefficient matrix after the above-mentioned quantization processing. The VLC module can convert the coefficient matrix after the above-mentioned quantization processing into a one-dimensional array through zigzag scanning when encoding. The tail of the one-dimensional array has a plurality of "0"s, which can be replaced by other forms. When decoding, the plurality of "0"s are restored to fill the 64 bits of the matrix. Therefore, the image data can be further compressed through VLC encoding. For example, 00000000 can be represented as 80, and restored to 00000000 when decoding.

[0064] For still images, the above-mentioned DCT module, quantizer and VLC module can complete the intra-frame encoding of image data and reduce the image data quantity.

[0065] A buffer is used to temporarily store compressed image data.

[0066] A rate control module is used to adjust the code rate of the image according to the data buffer amount of the buffer. When the image is more complex, the code rate is higher, and when the image is simpler, the code rate is lower. The rate control module is used to regulate the code rate within a certain range.

[0067] A dequantizer and an inverse DCT module are used to restore the encoded image data to the image data before encoding as a reference image frame.

[0068] A frame memory is used to store the reference frame and the motion vector.

[0069] A motion estimation and compensation module is used to calculate the motion vector and the motion residual according to the reference image frame and the current image frame.

[0070] For a moving image, inter-frame encoding of the image data can be performed according to the reference frame, the motion vector, and the motion residual to reduce the amount of image data.

[0071] Currently, in the process of inter-frame encoding of the image, the searched matching block can not be the best matching block, resulting in inaccurate motion vectors. Inaccurate motion vectors will result in a larger bit stream size and require a larger storage space. Moreover, inaccurate motion vectors will reduce the image quality. To solve this problem, the present application provides an image processing method that can find the best matching block and obtain more accurate motion vectors, thereby reducing the bit stream size and improving the image quality.

[0072] Figure 2 A flowchart of an image processing method provided by an embodiment of the present application is shown in FIG. 1, which includes steps 201-206. Figure 2

[0073] 201, obtain a current image frame and a reference image frame.

[0074] In combination with Figure 1 , the current image frame is an image frame received by an encoder, and the reference image frame is an image frame restored from the encoded image data by a dequantizer and an inverse DCT module.

[0075] In some embodiments, the current image frame can be an original current image frame or an image frame obtained by scaling the original current image frame at least once using a scaling factor K, and the reference image frame can be an original reference image frame or an image frame obtained by scaling the original reference image frame at least once using the scaling factor K. The present application does not limit whether the current image frame and the reference image frame are scaled or not, and does not limit the specific value of the scaling factor K. The following embodiments are exemplarily described with the scaling factor K equal to 2.

[0076] ​The number of times the current image frame and the reference image frame can be scaled depends on the image coding standard. For example, taking the Moving Picture Experts Group-2 (MPEG2) image coding standard as an example, since MPEG2 supports 8*8 image blocks, no scaling is performed on the current image frame and the reference image frame. In this case, the current image frame is the unscaled current image frame (also called the original current image frame), and the reference image frame is the unscaled reference image frame (also called the original reference image frame). Taking the High Efficiency Video Coding (HEVC) image coding standard as an example, since HEVC supports 4*4, 8*8, 16*16, 32*32, and 64*64 image blocks, a scaling factor of 2 can be used to scale the current image frame and the reference image frame four times, resulting in five levels of current image frames and reference image frames, with the size of these five levels decreasing sequentially. This application does not limit the specific coding standard used for image coding, the specific value of the scaling factor, or the specific number of scaling operations. For ease of explanation, the following example illustrates the scaling of the current image frame and the reference image frame twice, with a scaling factor of 2.

[0077] like Figure 3 As shown, taking a current image frame size of 1280*1280, a reference image frame size of 1280*1280, a scaling factor K of 2, and scaling twice as an example: First, using a scaling factor of 2, both the 1280*1280 current image frame and the reference image frame are scaled once, resulting in a 640*640 pixel current image frame and a 640*640 pixel reference image frame. Then, using the scaling factor of 2 again, both the 640*640 pixel current image frame and the 640*640 pixel reference image frame are scaled once, resulting in a 320*320 pixel current image frame and a 320*320 pixel reference image frame. That is, when the scaling number is 2, three levels of current image frame and reference image frame can be obtained. The first level is a current image frame of 1280*1280 pixels and a reference image frame of 1280*1280 pixels. The second level is a current image frame of 640*640 pixels and a reference image frame of 640*640 pixels. The third level is a current image frame of 320*320 pixels and a reference image frame of 320*320 pixels.

[0078] 202. The current image frame is downsampled and upsampled sequentially to obtain the processed current image frame. The reference image frame is downsampled and upsampled sequentially to obtain the processed reference image frame.

[0079] like Figure 4As shown, taking the current image frame and the reference image frame cannot be scaled as an example, the current image frame x n Down-sampling and up-sampling are sequentially performed to obtain the processed current image frame x n+1 , the reference image frame y n Down-sampling and up-sampling are sequentially performed to obtain the processed current image frame y n+1 .

[0080] In some embodiments, when the reference image frame is scaled for multiple times, due to the small size of the image of the last layer, in order to avoid the problem that image information may be lost after down-sampling and up-sampling processing of the small size image, the reference image frame obtained by scaling the most times can not be down-sampled and up-sampled, and the reference image frame obtained by scaling the most times is directly taken as the processed reference image frame. The embodiments of the present application do not limit whether the reference image frame obtained by scaling the most times is down-sampled and up-sampled, and the following embodiments are exemplarily described taking the reference image frame obtained by scaling the most times as an example.

[0081] As shown in Figure 5 , taking the current image frame as an image frame of 1280*1280 pixels, the reference image frame as an image frame of 1280*1280 pixels, and the scaling coefficient K as 2, scaling 2 times as an example. The current image frame p0 of 1280*1280 pixels is sequentially down-sampled and up-sampled to obtain the processed current image frame p1, the current image frame p n Down-sampling and up-sampling are sequentially performed to obtain the processed current image frame p n+1 , the current image frame p N Down-sampling and up-sampling are sequentially performed to obtain the processed current image frame p N+1 . The reference image frame q0 of 1280*1280 pixels is sequentially down-sampled and up-sampled to obtain the processed reference image frame q1, the reference image frame q n Down-sampling and up-sampling are sequentially performed to obtain the processed reference image frame q n+1 , the reference image frame q N Down-sampling and up-sampling are not performed, and the reference image frame q N is taken as the processed reference image frame q N+1 .

[0082] The image processing method provided in this application embodiment obtains current image frames and reference image frames of different sizes by scaling the current image frame and reference image frame at least once, and performs downsampling and upsampling processing on the current image frames and reference image frames of the corresponding sizes to reduce high-frequency parts that do not represent the main information in the current image frames and reference image frames, filter noise signals, and then obtain the best matching block for each current sub-image block at each size relatively accurately based on the processed current image frames and reference image frames.

[0083] 203. According to the preset division method, the processed current image frame and the processed reference image frame are divided into multiple current sub-image blocks and multiple reference sub-image blocks respectively.

[0084] Different preset partitioning methods can be used for different image coding standards. If an image coding standard supports one partitioning method, that partitioning method is used as the preset partitioning method. If an image coding standard supports multiple partitioning methods, the partitioning method with the fewest pixels can be used as the preset partitioning method. This application does not limit the specific partitioning method used as the preset partitioning method.

[0085] For example, taking MPEG2 as the image coding standard, since MPEG2 only supports 8x8 pixel image blocks, the default partitioning method is to use an 8x8 pixel size to divide the processed current image frame and the processed reference image frame into multiple current sub-image blocks and multiple reference sub-image blocks. Taking HEVC as the image coding standard, since HEVC supports 4x4, 8x8, 16x16, 32x32, and 64x64 pixel image blocks, the default partitioning method can use a 4x4 pixel size to divide the processed current image frame and the processed reference image frame into multiple current sub-image blocks and multiple reference sub-image blocks.

[0086] like Figure 6 As shown, taking HEVC as the encoding standard as an example, the processed current image frame can be divided into M current sub-image blocks and the processed reference image frame can be divided into M reference sub-image blocks using a 4*4 pixel preset division method.

[0087] The specific number of current sub-image blocks and the specific number of reference sub-image blocks are related to the image sizes of the current image frame and the reference image frame, as well as the preset division method. This application embodiment does not limit the specific number of current sub-image blocks and the specific number of reference sub-image blocks.

[0088] 204. Among multiple reference sub-image blocks, determine the reference sub-image block with the lowest similarity to each current sub-image block as the matching block for that current sub-image block.

[0089] Determining the matching block of the current sub-image block among multiple reference sub-image blocks may include steps 1-3.

[0090] Step 1: Within the search range corresponding to the current sub-image block, calculate the distances between the current sub-image block and multiple reference sub-image blocks within the search range.

[0091] Each current sub-image block corresponds to a search range. The search ranges corresponding to different sub-image blocks can be different, and the search ranges corresponding to different sub-image blocks can include the same image block. This application does not limit the size of the search range corresponding to each current sub-image block or the positional relationship between the search range and the current sub-image block. The size of the search range corresponding to the current sub-image block is related to the image encoding standard.

[0092] In some examples, taking HEVC as the image encoding standard and using a preset 4*4 pixel division method to divide the processed current image frame into multiple current sub-image blocks, the search range corresponding to the current sub-image block can be searched within the square surrounding the current sub-image block with a radius of 16 pixels or a radius of 32 pixels, etc.

[0093] In some embodiments, the distance between multiple reference sub-image blocks and the current sub-image block within the search range can be calculated using the mean squared error (MSE).

[0094] D i,j =MSE(K i Q j )

[0095] Among them, D i,j K represents the distance between the reference sub-image patch and the current sub-image patch. i Q represents the current sub-image patch. j This represents a reference sub-image block.

[0096] Combination Figure 6 ,like Figure 7 As shown, taking the current image frame after processing as an example, divided into M current sub-image blocks, Figure 7 The L reference sub-image blocks in the search are the image blocks within the search range corresponding to the current sub-image block. The distance between multiple reference sub-image blocks and the current sub-image block can be calculated through MSE. The closer the distance, the higher the similarity between the two image blocks.

[0097] Step 2: Regularize the distances between multiple reference sub-image blocks within the search range and the current sub-image block to obtain multiple candidate similarities.

[0098] The embodiments of this application do not limit the specific processing method of regularization. The smaller the similarity obtained by regularization, the smaller the difference between the reference sub-image block and the current sub-image block, and the more similar the reference sub-image block and the current sub-image block are.

[0099] In some examples, the distances between multiple reference sub-image patches and the current sub-image patch within the search range can be regularized using the following formula:

[0100]

[0101] Where S i,j D represents the similarity between the reference sub-image patch and the current sub-image patch. i,j This represents the distance between the reference sub-image patch and the current sub-image patch, where α is a non-zero parameter. D represents the minimum distance among multiple reference sub-image patches between the current sub-image patch and the reference sub-image patch. i,j .

[0102] like Figure 8 As shown, taking the current image frame after processing as divided into M current sub-image blocks, and the search range of the current sub-image block including L reference sub-image blocks as an example, multiple candidate similarities are obtained by regularizing the distance between multiple reference sub-image blocks and the current sub-image block.

[0103] Step 3: Determine the reference sub-image block corresponding to the smallest similarity among multiple candidate similarities as the matching block of the current sub-image block.

[0104] In some examples, the matching block of the current sub-image patch can be determined using the following formula:

[0105]

[0106] Where, j nn (i) represents the reference sub-image block (matching block) corresponding to the current sub-image block. This means finding the reference sub-image patch with the lowest similarity among multiple candidate similarities.

[0107] like Figure 9 As shown, taking the processed current image frame as divided into M current sub-image blocks, and the search range of each current sub-image block including L reference sub-image blocks as an example, through j nn (i) The reference sub-image block corresponding to the smallest similarity among multiple candidate similarities can be found, i.e., the reference sub-image block corresponding to the smallest similarity such as 0.7, 0.3 and 0.2 is the matching block of the current sub-image block.

[0108] The image processing method provided in the embodiments of the present application can ensure that the obtained matching block is the best matching block of the current image block by calculating the distance between the reference sub-image block and the current sub-image block, processing the distance, and determining the reference sub-image block corresponding to the minimum similarity as the matching block of the current sub-image block.

[0109] 205. Obtain a motion vector corresponding to the current sub-image block based on each current sub-image block and the matching block corresponding to the current sub-image block.

[0110] According to different image encoding standards, the current image frame and the reference image frame can be scaled for different times, and the number of motion vectors corresponding to the current sub-image block is also different. The embodiments of the present application are not limited to the specific number of motion vectors obtained according to the type of image encoding standard.

[0111] As shown in Figure 10 , the steps 203 and 204 can be performed by a patches nearest neighbors (PNN) module, and the specific execution steps of the PNN module can refer to the related content of the steps 203 and 204. Taking the image encoding standard as MPEG2 for example, the current image frame and the reference image frame are not scaled, and based on the current sub-image block and the matching block corresponding to the current sub-image block, one group of motion vectors mv=(c x , c y ) corresponding to the current sub-image block can be obtained, wherein (c x , c y ) represents the relative coordinates between the current image block and the matching block.

[0112] In some examples, if the current image frame and the reference image frame can be scaled for multiple times, under each image size, one motion vector corresponding to the current sub-image block can be obtained based on the current sub-image block and the matching block corresponding to the current sub-image block. Therefore, after the image is scaled for multiple times, multiple groups of motion vectors can be obtained for each current sub-image block:

[0113] mv=(K g c x , K g c y )

[0114] wherein K represents a scaling coefficient, and g represents different scaling levels corresponding to the current sub-image block.

[0115] As shown in Figure 11 , taking the current image frame and the reference image frame that can be scaled for 2 times, a total of 3 scaling levels, for example, for the first layer, the motion vector mv1=(K 1 c x , K 1 cy For the second layer, the motion vector mv2 = (K 2 c x K 2 c y For the third layer, the motion vector mv3 = (K 3 c x K 3 c y That is, in each layer, the motion vectors (c) between matching blocks of the current sub-image block are obtained. x c y After that, it is also necessary to change the motion vector (c) x c y Multiply by K g This yields the motion vector corresponding to the current image patch when it is uncompressed.

[0116] 206. Encode the current image frame based on motion vectors.

[0117] If a set of motion vectors corresponding to each current sub-image block and its corresponding matching block is obtained, the current image frame is encoded based on these motion vectors.

[0118] The image processing method provided in this application reduces high-frequency components that do not represent key information in the current and reference image frames by sequentially downsampling and upsampling the current and reference image frames, thus filtering out noise signals. Furthermore, by dividing the processed current and reference image frames into image blocks, the best matching block for each current sub-image block can be obtained more accurately. Therefore, the motion vector obtained based on the best matching block is more accurate. When encoding the current image frame based on the motion vector, the bitstream size can be reduced, thereby improving image quality.

[0119] like Figure 12 As shown, if multiple sets of motion vectors corresponding to the current sub-image blocks are obtained based on each current sub-image block and the matching block corresponding to the current sub-image block, the image processing method provided in this application embodiment includes steps 201-206 as described above, and may also include step 207 before step 206.

[0120] 207. Based on the first similarity corresponding to the first current sub-image block and the second similarity corresponding to the second current sub-image block, determine the target motion vector from the first motion vector corresponding to the first current sub-image block and the second motion vector corresponding to the second current sub-image block. Wherein, the first current sub-image block is an image block scaled down from the second current sub-image block.

[0121] In some examples, the first current sub-image block can be an image block after the second current sub-image block is scaled once, and the first current sub-image block can also be an image block after the second current sub-image block is scaled multiple times. Embodiments of the present application do not limit the number of times the first current sub-image block is scaled to the second current sub-image block.

[0122] The second current sub-image block can include multiple current sub-image blocks, and each current sub-image block can correspond to a second similarity. Embodiments of the present application do not limit the number of current sub-image blocks included in the second current sub-image block, and the number of current sub-image blocks included in the second current sub-image block is related to parameters such as the scaling factor.

[0123] For example, as shown in FIG. 6, taking the first current sub-image block as an image block in a current image frame of 320*320 pixels, the second current sub-image block as an image block in a current image frame of 640*640 pixels, and the size of the first current sub-image block and the second current sub-image block as 4*4 as an example, since the current image frame of 320*320 pixels is an image frame after the current image frame of 640*640 pixels is scaled once, the size of the current image frame of 320*320 pixels is smaller than that of the current image frame of 640*640 pixels, and therefore one 4*4 first current sub-image block in the current image frame of 320*320 pixels corresponds to four 4*4 second current sub-image blocks in the current image frame of 640*640 pixels. That is, the four 4*4 second current sub-image blocks can obtain one 4*4 first current sub-image block after being scaled once. Figure 3

[0124] The first similarity and the second similarity are compared, and in a case where the first similarity is less than or equal to the second similarity, the first motion vector is determined as the target motion vector. In a case where the first similarity is greater than the second similarity, the second motion vector is determined as the target motion vector.

[0125] As shown in FIG. 6, taking the scaling factor K as 2, the first current sub-image block including one sub-image block, the first similarity as S0, and the first current sub-image block as an image block after the second current sub-image block is scaled once as an example, the second current sub-image block including four sub-image blocks, and the second similarity including S1, S2, S3, and S4. The first similarity S0 and the second similarity S1-S4 are compared in turn, and in a case where the first similarity S0 is less than or equal to the second similarity S1-S4, the first motion vector is determined as the target motion vector. In a case where the first similarity S0 is greater than any second similarity in S1-S4, the second motion vector is determined as the target motion vector. Figure 13

[0126] ​​Correspondingly, the current image frame can be encoded based on the target motion vector determined in step 207 in step 206.

[0127] In some examples, the above steps 203-207 can be processed in parallel using a tensor processing framework to improve processing efficiency. For example, Pytorch (python torch), Tensorflow. The embodiments of the present application do not limit the type of tensor processing framework used for parallel computing.

[0128] In some examples, the encoding efficiency can be improved by using a graphics processing unit (GPU).

[0129] The image processing method provided by the embodiments of the present application can obtain a group of motion vectors and a similarity at each size when scaling the current image frame and the reference image frame, so it is necessary to compare the multiple groups of similarities of the current image block at different sizes and determine the best motion vector. Moreover, when determining the best motion vector, if the similarity of the large block is higher than that of the small block, the motion vector corresponding to the large block is taken as the best motion vector, and if the similarity of the large block is lower than that of the small block, the motion vector corresponding to the small block is taken as the best motion vector, so the best motion vector can be selected at different sizes, further improving the accuracy of the motion vector. In addition, the current image frame can be divided into image blocks of different sizes for encoding by the present solution. It can be understood that the smaller the similarity value is, the higher the similarity is, and the larger the similarity value is, the lower the similarity is.

[0130] The embodiments of the present application provide an image processing device, which can be an image encoder. Specifically, the image processing device is used to execute steps 201-207 in the above image processing method. The image processing device provided by the embodiments of the present application can include modules corresponding to the respective steps.

[0131] The embodiments of the present application can divide the image processing device into functional modules according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be realized in the form of hardware or software functional module. The division of modules in the embodiments of the present application is illustrative, and is only a logical functional division. When actually implemented, another division mode can be used.

[0132] In the case of dividing each functional module according to each function, Figure 14 A possible structure schematic diagram of the image processing device involved in the above embodiments is shown. As shown in Figure 14As shown, the image processing apparatus 1400 can include an acquisition module 1401, a sampling module 1402, a division module 1403, a determination module 1404, and an encoding module 1405. Specifically, the functions of the modules are as follows.

[0133] The acquisition module 1401 is configured to acquire a current image frame and a reference image frame.

[0134] The sampling module 1402 is configured to sequentially perform down-sampling and up-sampling on the current image frame to obtain a processed current image frame, and sequentially perform down-sampling and up-sampling on the reference image frame to obtain a processed reference image frame.

[0135] The division module 1403 is configured to divide the processed current image frame and the processed reference image frame into a plurality of current sub-image blocks and a plurality of reference sub-image blocks, respectively, according to a preset division manner.

[0136] The determination module 1404 is configured to determine, among the plurality of reference sub-image blocks, a reference sub-image block with a minimum similarity to each current sub-image block as a matching block of the current sub-image block.

[0137] The determination module 1404 is further configured to obtain a motion vector corresponding to each current sub-image block based on the current sub-image block and the matching block corresponding to the current sub-image block.

[0138] The encoding module 1405 is configured to encode the current image frame based on the motion vector.

[0139] In a possible implementation, the current image frame is an original current image frame or an image frame obtained by scaling the original current image frame at least once using a scaling factor K, and the reference image frame is an original reference image frame or an image frame obtained by scaling the original reference image frame at least once using the scaling factor K.

[0140] In a possible implementation, the determination module 1404 is specifically configured to: calculate distances between a plurality of reference sub-image blocks in a search range and the current sub-image block, respectively, within the search range corresponding to the current sub-image block. Regularize the distances between the plurality of reference sub-image blocks in the search range and the current sub-image block to obtain a plurality of candidate similarities. Determine, as the matching block of the current sub-image block, a reference sub-image block corresponding to a minimum similarity in the plurality of candidate similarities.

[0141] In a possible implementation, the minimum similarity in the plurality of candidate similarities corresponding to each current sub-image block is the similarity corresponding to the current sub-image block.

[0142] In an example implementation, the determining module 1404 is further configured to determine the target motion vector from the first motion vector corresponding to the first current sub-image block and the second motion vector corresponding to the second current sub-image block according to a first similarity corresponding to the first current sub-image block and a second similarity corresponding to the second current sub-image block. The first current sub-image block is the image block after scaling the second current sub-image block.

[0143] In an example implementation, the determining module 1404 is specifically configured to compare the first similarity and the second similarity, and determine the first motion vector as the target motion vector in a case that the first similarity is less than or equal to the second similarity. The second motion vector is determined as the target motion vector in a case that the first similarity is greater than the second similarity.

[0144] In an example implementation, the encoding module 1405 is specifically configured to encode the current image frame based on the target motion vector.

[0145] Some embodiments of the present disclosure provide a computer readable storage medium (for example, a non-transitory computer readable storage medium) having computer program instructions stored therein, which, when executed on a computer (for example, an image processing apparatus), cause the computer to perform the image processing method according to any one of the above embodiments.

[0146] In some examples, the above computer readable storage medium can include, but is not limited to, a magnetic storage device (for example, a hard disk, a floppy disk, or a magnetic tape, etc.), an optical disc (for example, a CD (Compact Disk), a DVD (Digital Versatile Disk), etc.), a smart card, and a flash memory device (for example, an EPROM (Erasable Programmable Read-Only Memory), a card, a stick, or a key drive, etc.). The various computer readable storage media described in the present disclosure can represent one or more devices and / or other machine readable storage media for storing information. The term "machine readable storage medium" can include, but is not limited to, a wireless channel and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0147] Some embodiments of the present disclosure also provide a computer program product, for example, stored on a non-transitory computer readable storage medium. The computer program product includes computer program instructions, which, when executed on a computer (for example, an image processing apparatus), cause the computer to perform the image processing method according to the above embodiments.

[0148] Some embodiments of the present disclosure also provide a computer program. When the computer program is executed on a computer (for example, an image processing apparatus), the computer program causes the computer to perform the image processing method as described in the above embodiments.

[0149] The computer readable storage medium, the computer program product and the computer program described above have the same beneficial effects as the image processing method described in some of the above embodiments, and will not be described here.

[0150] The above is merely specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any person skilled in the art can think of changes or replacements within the technical range disclosed by the present disclosure, which should be covered by the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the current image frame and the reference image frame; the current image frame is the image frame after scaling the original current image frame at least once using a scaling factor K, and the reference image frame is the image frame after scaling the original reference image frame at least once using the scaling factor K; The current image frame is downsampled and upsampled sequentially to obtain the processed current image frame. The reference image frame is downsampled and upsampled sequentially to obtain the processed reference image frame. According to a preset division method, the processed current image frame and the processed reference image frame are respectively divided into multiple current sub-image blocks and multiple reference sub-image blocks; Within the search range corresponding to the current sub-image block, calculate the distances between the current sub-image block and multiple reference sub-image blocks within the search range; The distances between the multiple reference sub-image blocks within the search range and the current sub-image block are regularized to obtain multiple candidate similarities; the minimum similarity among the multiple candidate similarities corresponding to each current sub-image block is the similarity corresponding to that current sub-image block. The reference sub-image block corresponding to the smallest similarity among the multiple candidate similarities is determined as the matching block of the current sub-image block; Based on each current sub-image block and the matching block corresponding to the current sub-image block, the motion vector corresponding to the current sub-image block is obtained; The current image frame is encoded based on the motion vector; Based on a first similarity corresponding to a first current sub-image block and a second similarity corresponding to a second current sub-image block, a target motion vector is determined from a first motion vector corresponding to the first current sub-image block and a second motion vector corresponding to the second current sub-image block; wherein, the first current sub-image block is an image block scaled down from the second current sub-image block; the step of determining the target motion vector from the first motion vector corresponding to the first current sub-image block and the second motion vector corresponding to the second current sub-image block based on the first similarity corresponding to the first current sub-image block and the second motion vector corresponding to the second current sub-image block includes: Comparing the first similarity and the second similarity, if the first similarity is less than or equal to the second similarity, the first motion vector is determined as the target motion vector; if the first similarity is greater than the second similarity, the second motion vector is determined as the target motion vector.

2. The method according to claim 1, characterized in that, The encoding of the current image frame based on the motion vector includes: The current image frame is encoded based on the target motion vector.

3. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the current image frame and the reference image frame; the current image frame is an image frame after scaling the original current image frame at least once using a scaling factor K, and the reference image frame is an image frame after scaling the original reference image frame at least once using the scaling factor K; The sampling module is used to sequentially downsample and upsample the current image frame to obtain the processed current image frame, and sequentially downsample and upsample the reference image frame to obtain the processed reference image frame. The partitioning module is used to divide the processed current image frame and the processed reference image frame into multiple current sub-image blocks and multiple reference sub-image blocks according to a preset partitioning method; The determination module is used for: Within the search range corresponding to the current sub-image block, calculate the distances between the current sub-image block and multiple reference sub-image blocks within the search range; The distances between the multiple reference sub-image blocks within the search range and the current sub-image block are regularized to obtain multiple candidate similarities; the minimum similarity among the multiple candidate similarities corresponding to each current sub-image block is the similarity corresponding to that current sub-image block. The reference sub-image block corresponding to the smallest similarity among the multiple candidate similarities is determined as the matching block of the current sub-image block; The determining module is further configured to obtain the motion vector corresponding to the current sub-image block based on each current sub-image block and the matching block corresponding to the current sub-image block; The encoding module is used to encode the current image frame based on the motion vector; The determining module is further configured to determine a target motion vector from the first motion vector corresponding to the first current sub-image block and the second motion vector corresponding to the second current sub-image block based on the first similarity corresponding to the first current sub-image block and the second similarity corresponding to the second current sub-image block; wherein, the first current sub-image block is an image block scaled up from the second current sub-image block; The determining module is specifically used to compare the first similarity and the second similarity. If the first similarity is less than or equal to the second similarity, the first motion vector is determined as the target motion vector; if the first similarity is greater than the second similarity, the second motion vector is determined as the target motion vector.

4. The apparatus according to claim 3, characterized in that, The encoding module is specifically used to encode the current image frame based on the target motion vector.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed on a computer, cause the computer to perform the image processing method as described in claim 1 or 2.

6. A computer program product, characterized in that, The computer program product includes computer program instructions that, when executed on a computer, cause the computer to perform the image processing method as described in claim 1 or 2.

Images