Motion vector determination method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a motion vector determination method and device, equipment, a storage medium and a product, and the method comprises the steps: determining the size of an interpolation output image corresponding to a to-be-processed image according to the size of the to-be-processed image, and enabling the size of the interpolation output image to be larger than the size of the to-be-processed image; performing first interpolation processing on each first type of block in the to-be-processed image to obtain a first interpolation result, the size of the first interpolation result being larger than the size of the first type of block, and the size of the first interpolation result being smaller than or equal to the size of a corresponding block in the interpolation output image; and performing first bidirectional predictive coding processing according to the first interpolation result so as to determine a motion vector corresponding to the to-be-processed block taking the first type block as the reference. According to the technical scheme provided by the embodiment of the invention, when the sub-blocks in the image are subjected to interpolation processing, the situation that the sub-blocks with relatively large sizes in the interpolation result cover the surrounding sub-blocks can be avoided, and the accuracy of coding and decoding can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to the technical field of multimedia data processing, and more particularly to a method, apparatus, device, storage medium, and product for determining a motion vector. Background Art

[0002] With the continuous development of digital technology, video, with its intuitive and rich content, has become increasingly widely used in our daily lives, such as short videos, video conferencing, and live broadcasts. The encoder can compress and encode the video to reduce the amount of data required to represent it. The decoder can decode the video bitstream to obtain the video transmitted by the encoder. Inter-frame prediction reduces video redundancy by exploiting the temporal correlation between consecutive frames. To improve prediction accuracy, sub-pixel motion vectors can be used. Through interpolation techniques, sub-pixel precision can obtain more precise motion information than full-pixel precision, thereby improving prediction accuracy.

[0003] During the interpolation process, the sizes of the output sub-blocks obtained from different interpolation operations may vary. Sometimes, the output sub-block obtained from the interpolation process is larger than the input sub-block. Therefore, when saving the interpolation result, the larger output sub-block may overlap the surrounding output sub-blocks, resulting in incorrect encoding and decoding. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, device, storage medium, and product for determining a motion vector, which can avoid the situation where larger sub-blocks cover other sub-blocks when saving the interpolation results when interpolating sub-blocks in an image, thereby improving the accuracy of encoding and decoding.

[0005] In a first aspect, an embodiment of the present application provides a method for determining a motion vector, including:

[0006] determining the size of an interpolation output image corresponding to the image to be processed according to the size of the image to be processed, wherein the size of the interpolation output image is larger than the size of the image to be processed;

[0007] performing a first interpolation process on each first-type block in the image to be processed to obtain a first interpolation result, where the size of the first interpolation result is larger than the size of the first-type block and the size of the first interpolation result is smaller than or equal to the size of the corresponding block in the interpolation output image;

[0008] A first bidirectional predictive coding process is performed according to the first interpolation result to determine a motion vector corresponding to a block to be processed with the first type of block as a reference.

[0009] In a second aspect, an embodiment of the present application provides a device for determining a motion vector, including:

[0010] a determining unit, configured to determine a size of an interpolated output image corresponding to the image to be processed according to a size of the image to be processed, wherein the size of the interpolated output image is larger than the size of the image to be processed;

[0011] a processing unit configured to perform a first interpolation process on each first-type block in the image to be processed to obtain a first interpolation result, wherein the size of the first interpolation result is larger than the size of the first-type block and the size of the first interpolation result is smaller than or equal to the size of the corresponding block in the interpolation output image; and perform a first bidirectional predictive coding process based on the first interpolation result to determine a motion vector corresponding to the block to be processed with the first-type block as a reference.

[0012] In a third aspect, an embodiment of the present application provides a device for determining a motion vector, which includes one or more processors; a memory for storing one or more computer programs, and when the one or more computer programs are executed by the one or more processors, the electronic device implements the method for determining a motion vector of the first aspect mentioned above.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the method for determining the motion vector of the first aspect.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, the method for determining the motion vector as in the first aspect is implemented.

[0015] In the technical solutions provided in some embodiments of the present application, after determining the size of an interpolated output image larger than the size of the image to be processed based on the size of the image to be processed, a first interpolation process can be performed on each first-type block in the image to be processed to obtain a first interpolation result, wherein the size of the first interpolation result is larger than the size of the first-type block and equal to or greater than the size of the corresponding block in the interpolated output image. A first bidirectional predictive coding process can then be performed based on the first interpolation result to determine the motion vector corresponding to the block to be processed with the first-type block as a reference. Thus, by determining a larger interpolated output image size, the size of each block in the interpolated output image can be larger than the size of the first interpolation result corresponding to each first-type block. When storing the first interpolation result larger than the size of the first-type block, the situation of overlapping the interpolation results of other sub-blocks can be avoided to a certain extent, thereby improving the accuracy of encoding and decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 1 is a schematic diagram of the architecture of a motion vector determination system provided in an embodiment of the present application;

[0018] Figure 2 It is a flow chart of a video encoder;

[0019] Figure 3 It is a structural diagram of block division;

[0020] Figure 4 It is a schematic diagram of the principle of inter-frame prediction;

[0021] Figure 5 1 is a flow chart of a method for determining a motion vector provided in an embodiment of the present application;

[0022] Figure 6 This is a schematic diagram of determining the size of an interpolated output image provided by an embodiment of the present application;

[0023] Figure 7 This is a schematic diagram of the principle of storing an interpolation result in an output block provided by an embodiment of the present application;

[0024] Figure 8 It is a schematic diagram of the principles of building and determining candidate MVPs;

[0025] Figure 9 This is a schematic diagram of an application of a method for determining a motion vector provided by an embodiment of the present application in a live video broadcasting scenario;

[0026] Figure 10 This is a schematic diagram of an application of a method for determining a motion vector provided by an embodiment of the present application in a video-on-demand scenario;

[0027] Figure 11 1 is a schematic structural diagram of a device for determining a motion vector provided in an embodiment of the present application;

[0028] Figure 12 It is a structural diagram of a motion vector determination device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] It should be noted in advance that, in order to enable those skilled in the art to better understand the technical solutions proposed in the embodiments of the present application, the embodiments of the present application will be combined with one or more drawings to clearly and completely describe the implementation of the technical solutions proposed in the embodiments of the present application. In addition, the various drawings shown in the embodiments of the present application are only exemplary illustrations. For example, the execution order of the various steps in the drawings can be adaptively adjusted according to the actual application scenario. In addition, in the embodiments of the present application, the block diagrams shown in the drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or these functional entities can be implemented in one or more hardware modules or integrated circuits, or these functional entities can be implemented in different networks and / or processor devices and / or microcontroller devices.

[0030] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0031] It should be noted that the term "plurality" used in this document refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0032] The embodiments of the present application relate to multimedia data processing technology, and specifically to the process of encoding and decoding video content. The video content mentioned in the embodiments of the present application may refer to video content that is generated in real time and needs to be transmitted in real time during live video broadcasts, video conferences, etc., and the video content may also refer to video content that does not need to be generated and transmitted in real time in scenarios such as video on demand and short video content. Among them, the so-called video encoding refers to the technology of encoding video content, converting the video content into a smaller file size, and obtaining the video code stream corresponding to the video content. The so-called video decoding refers to the technology of decoding the video code stream, which is the inverse process of video encoding, and restores the video code stream to uncompressed video content to obtain the video content corresponding to the video code stream.

[0033] The video stream processing solution proposed in the embodiments of this application involves cloud storage and other technologies, among which:

[0034] Cloud storage is a new concept that has been extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0035] Currently, storage systems utilize a method for creating logical volumes. When creating a logical volume, physical storage space is allocated for each logical volume. This physical storage space may consist of disks on a specific storage device or several storage devices. When a client stores data on a logical volume, it stores the data on a file system. The file system divides the data into multiple parts, each of which is an object. An object contains not only the data but also additional information such as the data identifier. The file system writes each object to the physical storage space of the logical volume and records the storage location of each object. Therefore, when a client requests access to data, the file system can provide access based on the storage location of each object.

[0036] The storage system allocates physical storage space to logical volumes by pre-dividing the physical storage space into stripes based on the estimated capacity of the objects to be stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the Redundant Array of Independent Disks (RAID) groupings. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0037] See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of a motion vector determination system provided by an embodiment of the present application. Figure 1As shown, the motion vector determination system includes a video encoding device 101 and a video decoding device 102, and the video encoding device 101 and the video decoding device 102 are video processing devices. Among them, the video encoding device 101 can be a video processing device for producing video content. After producing the video content, the video content can be encoded to obtain a video code stream. Furthermore, the video encoding device 101 can transmit the video code stream to the video decoding device 102. The video decoding device 102 can be a video processing device that receives the video code stream sent by the video encoding device 101 and decodes the video code stream to obtain the video content corresponding to the video code stream. The video encoding device 101 can be an electronic device used by a provider of video content (such as a producer of video content). The video decoding device 102 can be an electronic device used by a user of video content (such as a user).

[0038] The video encoding device 101 and the video decoding device 102 can be connected directly or indirectly via a wired or wireless method, for example Figure 1 As shown, the video encoding device 101 and the video decoding device 102 can communicate via a network 103. The video encoding device 101 and the video decoding device 102 can perform one-way data transmission over the network 103. For example, the video encoding device 101 can encode video content and transmit it to the video decoding device 102 over the network 103. The encoded video content is transmitted in the form of a video stream. The video decoding device 102 can obtain the video stream over the network 103, decode the video stream to recover the video content, and then display the recovered video content.

[0039] In another exemplary embodiment, the video encoding device 101 and the video decoding device 102 can perform bidirectional data transmission based on the network 103. The bidirectional data transmission can occur during live video broadcasting, video conferencing, and the like. For bidirectional data transmission, the video encoding device 101 can encode video content (e.g., a video picture stream captured by the video encoding device 101) in real time, and then transmit the encoded video code stream to the video decoding device 102 in real time via the network 103. The video decoding device 102 can receive the video code stream transmitted by the video encoding device 101, decode the video code stream to restore the video content, and then display the video pictures in the video picture stream on an accessible display device according to the video content.

[0040] It should be noted that Figure 1The number and form of devices shown are for example purposes only and do not constitute a limitation on the embodiments of the present application. In actual applications, the video processing device (such as the video encoding device 101 and the video decoding device 102) can be the same electronic device or different electronic devices, and this application does not limit this. Among them, the above-mentioned network 103 can include, for example, a wired and / or wireless communication network. The communication network 150 can exchange data in a circuit switching and / or packet switching channel. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of the network 150 may be irrelevant to the operations disclosed in this application.

[0041] The video processing devices (such as the video encoding device 101 and the video decoding device 102) can be, but are not limited to, smartphones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, smart voice interaction devices, smart home appliances, vehicle-mounted terminals, VR devices, aircraft, etc. The video processing devices (such as the video encoding device 101 and the video decoding device 102) can also be servers, for example, independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0042] The processing process performed by the video encoding device 101 mainly includes: (1) the process of obtaining and producing video content; (2) the process of video encoding. The processing process performed by the video decoding device 102 mainly includes (3) the process of video decoding. Among them, the video encoding device 101 and the video decoding device 102 involve the transmission process of the video code stream. The transmission process can be carried out based on various transmission protocols. The transmission protocols here may include but are not limited to: Dynamic Adaptive Streaming over HTTP (DASH) protocol, Dynamic Bitrate Adaptive Transmission (HTTP Live Streaming, HLS) protocol, Smart Media Transport Protocol (SMTP), Transmission Control Protocol (TCP), etc.

[0043] Please also refer to Figure 2 , Figure 2 It is a flow chart of a video encoder, combined with Figure 2The various processes involved in the video encoding device 101 and the video decoding device 102 are respectively introduced in detail:

[0044] (1) The process of acquiring and producing video content

[0045] Video content can be acquired through two methods: capturing real-world sound and visual scenes through a capture device, and generating it through electronic devices. Specifically, the capture device can refer to hardware components configured in the video encoding device 101. For example, the capture device can include a microphone, a camera, and various sensors, such as an activated radar sensor. The capture device can also be a device directly or indirectly connected to the video encoding device 101 via wired or wireless means. For example, if the video encoding device 101 is a server, the capture device can be a camera connected to the server, providing video content acquisition functionality for the video encoding device 101. The capture device can include a camera and a sensor device, and can also include an audio device for acquiring audio content synchronized with the video content. For example, the camera device can include a standard camera, a depth camera, a light field camera, etc. The sensor device can include a laser device, a radar device, etc. The audio device can include an audio sensor, a microphone, etc. The capture device is deployed at a specific location in real space to capture the video content within that space and the audio content synchronized with the video content.

[0046] (2) Video encoding process

[0047] The video encoding device 101 performs video encoding on the available video signal (video content). The video signal (video content) can be understood as video data in a color mode (Red, Green, Blue, RGB) / luminance-bandwidth-chrominance (YUV) video data. Taking the international video coding standard (High Efficiency Video Coding, HEVC), the international video coding standard (Versatile Video Coding, VVC), and the Chinese national video coding standard (Audio Video Coding Standard, AVS) as examples, a hybrid coding framework is adopted to perform the following series of operations and processing on the input original video signal:

[0048] 1) Image Partition / Block Partition Structure

[0049] After a video frame image is input, it will be divided into several non-overlapping processing units according to a block size, and each processing unit will perform similar compression operations. This processing unit is called a coding tree unit (CTU), or the largest coding unit (LCU). CTU is used to divide the image into independently processable blocks, thereby improving coding efficiency and parallelism. In VVC, the CTU size is 128x128, and it can be further divided into more refined parts to obtain one or more basic coding units (CU). CU is the most basic element in a coding link. VVC supports quadtree, binary tree, and ternary tree partitioning. For example, please refer to Figure 3 , Figure 3 It is a block division structure diagram, such as Figure 3 As shown, a CTU can be divided downward in a quadtree manner.

[0050] 2) Predictive Coding

[0051] Predictive coding includes intra-frame prediction and inter-frame prediction. The original video signal is predicted by the selected reconstructed video signal to obtain a residual video signal. The encoder needs to determine which predictive coding mode to use for the current coding unit (or coding block) and inform the video decoding device 102.

[0052] a. Intra-frame prediction means that the predicted signal comes from the area that has been encoded and reconstructed within the same image.

[0053] b. Inter-frame prediction means that the predicted signal comes from another image that has already been encoded and is different from the current image (called a reference image). This is because adjacent video frames usually have great similarities. For the image block to be encoded (such as the CU mentioned above), a reference block can be searched in the reference image, and their image residual and position information can be recorded. The position information can be represented by the difference in the upper left corner coordinates between the co-located block of the image block in the reference image and the reference block. This position information can be called a motion vector (MV), a motion vector, a displacement vector, etc. The process of searching for the optimal reference block is called motion estimation (ME), and the process of reconstructing the current block using the reference block is called motion compensation (MC).

[0054] MC is a commonly used prediction method in video coding. Based on the redundant characteristics of video content in the temporal or spatial domain, MC derives the predicted value of the current coding block from the coded area. This type of prediction method includes inter-frame prediction, intra-frame block copy prediction, intra-frame string copy prediction, etc. In specific coding implementations, these prediction methods may be used alone or in combination. For coding blocks using these prediction methods, it is usually necessary to explicitly or implicitly encode one or more two-dimensional displacement vectors in the bitstream to indicate the displacement of the current block (or the current block's co-located block) relative to one or more reference blocks.

[0055] Please also refer to Figure 4 , Figure 4 It is a schematic diagram of the principle of inter-frame prediction, such as Figure 4 As shown in Figure 1, inter-frame prediction uses the correlation of the video time domain to use the pixels of the adjacent coded images to predict the pixels of the current image, so as to effectively remove the redundancy of the video time domain and effectively save the bits of the coded residual data. Wherein, P represents the current frame, Pr represents the reference frame, B represents the current coding block, and Br represents the reference block of B. The coordinates of B' in the reference frame are the same as the coordinates of B in the current frame. The coordinates of Br are (x r ,y r ), the coordinates of B' are (x, y), and the displacement between the current coding block and its reference block is MV, where MV=(x r -x,y r -y).

[0056] In order to improve the prediction accuracy, sub-pixel precision MV can be used for prediction. For example, in video content, the movement of objects is not necessarily based on whole pixels as the basic unit. Therefore, the movement of less than one pixel can be represented by sub-pixel precision MV. For example, in VVC, the highest MV that can be used is 1 / 16 precision, which means that the movement of 1 / 16 pixels can be represented. In order to obtain MV that can represent sub-pixel precision, the current coding block can be interpolated on the reference block to obtain a sub-pixel image. The pixel points obtained by the interpolation process are obtained by weighted averaging based on the filtering algorithm. The specific process can be to first perform half-pixel (i.e., 1 / 2 pixel) interpolation on the reference block, and then perform 1 / 4 pixel interpolation until the preset prediction accuracy or implementation accuracy is reached.

[0057] 3) Transform & Quantization

[0058] After the residual video signal undergoes transformations such as the Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT), it is converted to a transform domain, known as transform coefficients. The transform coefficients are then subjected to a lossy quantization operation, which loses some information, making the quantized signal more suitable for compression. In some video coding standards, more than one transform scheme may be available, so the encoder must select one for the current coding unit (or coding block) and inform the decoder. The level of quantization is typically determined by the quantization parameter (QP). A larger QP value means that coefficients with a larger value range will be quantized to the same output, which generally results in greater distortion and a lower bitrate. Conversely, a smaller QP value means that coefficients with a smaller value range will be quantized to the same output, which generally results in less distortion and a higher bitrate.

[0059] 4) Entropy Coding or Statistical Coding

[0060] The quantized transform domain signal is statistically compressed and encoded based on the frequency of occurrence of each value, ultimately outputting a binary (0 or 1) compressed bitstream. At the same time, the encoding generates other information, such as the selected coding mode and motion vector data, which also requires entropy coding to reduce the bitrate. Statistical coding is a lossless encoding method that can effectively reduce the bitrate required to express the same signal. Common statistical coding methods include variable length coding (VLC) and context-based binary arithmetic coding (CABAC).

[0061] The context-based binary arithmetic coding (CABAC) process consists of three main steps: binarization, context modeling, and binary arithmetic coding. After binarization of the input syntax elements, the binary data can be encoded using either the normal coding mode or the bypass coding mode. The bypass coding mode eliminates the need to assign a specific probability model to each binary bit. Instead, the input binary bit bin values ​​are directly encoded using a simple bypass encoder, speeding up both encoding and decoding. Generally, different syntax elements are not completely independent, and even the same syntax elements have some memory. Therefore, according to conditional entropy theory, conditional coding using other coded syntax elements can further improve coding performance compared to independent encoding or memoryless coding. This coded symbol information used as a condition is called context. In the normal coding mode, the binary bits of the syntax elements are sequentially fed into the context modeler. The encoder assigns an appropriate probability model to each input binary bit based on the values ​​of previously coded syntax elements or binary bits. This process is known as context modeling. The context model corresponding to the syntax element can be located using the context index increment (ctxIdxInc) and the context start index (ctxIdxStart). After the bin value and the assigned probability model are fed into the binary arithmetic encoder for encoding, the context model needs to be updated based on the bin value, which is the adaptive process in encoding.

[0062] 5) Loop Filtering

[0063] The changed and quantized signal will be reconstructed through inverse quantization, inverse transformation and prediction compensation operations to obtain a reconstructed image. Compared with the original image, due to the influence of quantization, some information of the reconstructed image is different from the original image, that is, the reconstructed image will produce distortion (Distortion). Therefore, the reconstructed image can be filtered, such as deblocking filter (DB), adaptive pixel compensation (Sample Adaptive Offset, SAO) or adaptive loop filter (ALF) and other filters, which can effectively reduce the degree of distortion caused by quantization. Since these filtered reconstructed images will be used as a reference for subsequent encoded images to predict future image signals, the above filtering operation is also called loop filtering, that is, filtering operation within the encoding loop.

[0064] Current mainstream video coding standards, such as HEVC, VVC, AV1 (the first-generation video coding standard developed by the Alliance for Open Media), AV2 (the second-generation video coding standard developed by the Alliance for Open Media), and AVS3 (the third-generation video coding standard developed by the Alliance for Open Media), all use a block-based hybrid coding framework. Specifically, the original video content is divided into a series of coding blocks and combined with video coding methods such as prediction, transform, and entropy coding to achieve video data compression.

[0065] In one embodiment of the present application, Figure 2 The basic flow chart of a video encoder is shown in FIG. In this flow chart, intra-frame prediction is used as an example. k [x,y] and predicted image signal Perform difference operation to obtain the residual signal u k [x,y], residual signal u k [x,y] is transformed and quantized to obtain the quantized coefficients. The quantized coefficients are entropy coded to obtain the encoded bit stream, and the reconstructed residual signal u' is obtained by inverse quantization and inverse transformation. k [x,y], predicted image signal and the reconstructed residual signal u' k [x,y] superposition generates image signal Image signal On the one hand, it is input to the intra-frame mode decision module and the intra-frame prediction module for intra-frame prediction processing, and on the other hand, the reconstructed image signal s' is output through loop filtering. k [x,y], reconstructed image signal s' k [x,y] can be used as the reference image for the next frame for motion estimation and motion compensation prediction. Then based on the result of motion compensation prediction s' r [x+m x ,y+m y ] and intra prediction results Get the predicted image signal of the next frame And continue to repeat the above process until the encoding is completed.

[0066] (3) Video decoding process

[0067] The decoding process of the video decoding device 102 is the opposite of the encoding process of the video encoding device 101. According to the above video encoding process, when the video decoding device 102 performs video decoding, for each CU, the content playback device obtains the compressed code stream and first performs entropy decoding to obtain various mode information (MV, motion estimation information, etc.) and quantized transform coefficients. Each coefficient is dequantized and inversely transformed to obtain a residual signal (residual data). On the other hand, based on the known encoding mode information, the prediction signal corresponding to the CU can be obtained. After adding the two, the reconstructed signal can be obtained. Finally, the reconstructed value of the decoded image needs to undergo a loop filtering operation to produce the final output signal, which is the restored video content.

[0068] During the interpolation process, it was found that the sizes of the sub-blocks of the interpolation results obtained by different interpolation processes may be different. In certain modes, the size of the output sub-block obtained by the interpolation process may be larger than the size of the original reference block. Therefore, when saving the interpolation result, the larger output sub-block may overlap the surrounding output sub-blocks, resulting in incorrect encoding and decoding. In an embodiment of the present application, during the interpolation process, the video processing device (such as the above-mentioned video encoding device 101 and video decoding device 102) can determine the size of the interpolated output image corresponding to the image to be processed based on the size of the image to be processed. The size of the interpolated output image is larger than the size of the image to be processed. Then, the video processing device can perform a first interpolation process on each first type block in the video to be processed to obtain a first interpolation result. The size of the first interpolation result is larger than the size of the first type block, and the size of the first interpolation result is less than or equal to the size of the corresponding block in the interpolation output image. Finally, the video processing device can perform a first bidirectional predictive coding process based on the first interpolation result to determine the motion vector corresponding to the block to be processed with each first type block as a reference.

[0069] Therefore, by determining the size of a larger interpolation output image, the size of each block in the interpolation output image can be larger than the size of the first interpolation result corresponding to each first-type block. This is beneficial for avoiding overwriting the interpolation results of other sub-blocks when storing the first interpolation result that is larger than the size of the first-type block, thereby improving the accuracy of encoding and decoding.

[0070] In one implementation, the aforementioned video stream and video content (each video frame) can be stored on a blockchain, preventing tampering. Blockchain is a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. It is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of the information (to prevent counterfeiting) and generate the next block.

[0071] It can be understood that the motion vector determination system described in the embodiment of the present application is for the purpose of more clearly illustrating the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0072] Based on the above-mentioned motion vector determination system, the embodiment of the present application provides a method for determining a motion vector. The method for determining a motion vector described in the embodiment of the present application can be executed by an electronic device, which can be Figure 1 In the video processing device (such as the video encoding device 101 and the video decoding device 102) in the motion vector determination system shown, the video encoding device 101 and the video decoding device 102 can be the same electronic device. Figure 5 , Figure 5 : is a flow chart of a method for determining a motion vector provided by an embodiment of the present application. The processing of the video stream includes the following steps S501 to S503:

[0073] S501 : Determine the size of an interpolated output image corresponding to the image to be processed according to the size of the image to be processed.

[0074] In the embodiment of the present application, the image to be processed is an encoded or decoded video frame (image frame). It can be understood that for the encoder, inter-frame prediction uses the correlation in the video time domain to find a block in the encoded video frame that is very close to the current encoded block as a prediction block. This block with a very small difference can be called a reference block, and this encoded video frame can be called a reference frame, which is the image to be processed in the embodiment of the present application. For the decoder, the prediction block is found in the decoded video frame and then used as the prediction block of the current decoded block, and then the current decoded block can be reconstructed based on the prediction block and the residual data. The decoded video frame is the reference frame, which is the image to be processed in the embodiment of the present application.

[0075] Among them, the size of the image to be processed is the size of the image to be processed. The interpolation output image corresponding to the image to be processed is an image used to store the interpolation results corresponding to each block in the image to be processed. The interpolation output image includes output blocks corresponding to each block in the image to be processed, that is, the interpolation results corresponding to each block. The size of the interpolation output image is the size of the interpolation output image. Due to the existence of a specific mode, the size of the interpolation result (output block) obtained by the interpolation process may be larger than the size of the original reference block. Therefore, the size of the interpolation output image is larger than the size of the image to be processed. The image to be processed includes multiple blocks, each of which can be called an output block, and each block can be used as the interpolation result of the reference block corresponding to the block to be processed. The blocks in the image to be processed can be classified according to different types of interpolation processing, and different types of interpolation processing correspond to different inter-frame prediction coding modes.

[0076] It should be noted that during video encoding, considering the strong correlation between adjacent blocks in the temporal or spatial domains, MV prediction technology can be used to further reduce the bits required to encode the MV. In some codec standards, inter-frame prediction includes merging. Merge mode is a motion vector prediction technology. Specifically, it refers to an inter-frame coding mode that directly uses the motion information (i.e., MV) of adjacent blocks as candidate predicted motion vectors (MVP) and takes the best one as the motion information of the current block. That is, for a coding block using Merge mode, its MV is equal to the MVP.

[0077] In Merge mode, the encoder creates a candidate list of MVs for the current prediction unit (PU) (referred to as the Merge candidate list, Merge list, etc.), iterates over the candidate MVs in the list, and selects the one with the lowest rate distortion optimization (RDO) cost as the optimal MV. If the decoder creates the candidate list in the same way as the encoder, it will obtain the same Merge candidate list. Therefore, the encoder only needs to transmit the index of the optimal MV in the candidate list, reducing the codewords required to transmit the MV and enabling more efficient MV encoding.

[0078] Among them, the decoder can divide the decoded video frames into two groups, namely the reference frame lists L0 and L1. These decoded video frames can be used as reference frames to predict the decoded blocks in the current frame (which can be understood as the current block). The video frames in the reference frame list L0 are frames that are played before the current frame in the playback order and are called forward reference frames. The video frames in the reference frame list L1 are frames that are played after the current frame in the playback order and are called backward reference frames. The reference frame indexes refIdxL0 and refIdxL1 are used to indicate which two reference frames in L0 and L1 are used to predict the current block. In the case of bidirectional prediction, two prediction blocks will be obtained from L0 and L1 through MV0 and MV1 respectively. Among them, the MV0 can be obtained by the decoder based on the constructed forward Merge candidate list (list0) and index, and the MV1 can be obtained by the decoder based on the constructed backward Merge candidate list (list1) and index.

[0079] Furthermore, the decoder can combine the two prediction blocks obtained separately to obtain a prediction block (prediction signal) for the current block. If the current block is encoded based on the Merge mode, the reference frame indexes refIdxL0 and refIdxL1 and motion vectors MV0 and MV1 of the adjacent blocks will be directly used to obtain the block (prediction signal) of the current block. In this case, the bit rate of encoding motion information will be very low, and no additional information needs to be explicitly encoded, but it may cause inaccurate prediction problems.

[0080] Decoder-Side Motion Vector Refinement (DMVR) is a technology proposed to improve the accuracy of bidirectional prediction in Merge mode. DMVR does not directly use the above-mentioned MV0 and MV1, but searches for more accurate MV0' and MV1' near MV0 and MV1. In detail, assuming that the MVs obtained by Merge mode can be called Initial MVs, such as MV0 and MV1, which are MVs with integer pixel accuracy, DMVR can refine the Initial MVs through block matching. In the two reference frames, it traverses and searches the surrounding blocks of the block pointed to by the Initial MVs. The two most matching blocks will be used to generate the final prediction block, and the new MVs formed are called refined MVs.

[0081] The DMVR search process includes both whole-pixel and sub-pixel searches. To perform the sub-pixel search, DMVR typically requires interpolation of the reference block to generate sub-pixel predictions. During interpolation, pixels at the edges of the reference block are typically expanded. This is because the interpolation algorithm needs to access the neighboring pixels of the reference block edge pixels to calculate the new pixel values. Expanding edge pixels ensures that the interpolation algorithm correctly handles boundary regions and avoids interpolation errors caused by missing neighboring pixels. Therefore, if each side is expanded by R pixels in the DMVR interpolation process, the size of the original reference block (W) × (H) can be expanded to (W + 2R) × (H + 2R). For example, if the original reference block is 16 × 16, and each side is expanded by 2 pixels, with two rows or columns of pixels on each of the top, bottom, left, and right sides, the resulting interpolated block size is 20 × 20. Therefore, this type of interpolation processing can be called DMVR interpolation. In the embodiment of the present application, the DMVR interpolation processing can be called the first interpolation processing, and the blocks that need to be DMVR interpolated in the reference frame can be called the first type of blocks.

[0082] The encoder can also use other modes to improve MV accuracy and coding efficiency, such as Adaptive Motion Vector Resolution (AMVR). This means that the encoder can adaptively select the accuracy of motion vectors based on the scene content. During the encoding process, the encoder can dynamically choose to use higher motion vector accuracy (such as 1 / 8 pixel or 1 / 16 pixel) to improve prediction accuracy, or use lower accuracy to reduce coding overhead when the scene content is relatively simple. In AMVR, to generate sub-pixel precision prediction values, AMVR also interpolates the reference block. AMVR interpolation does not require expanding the pixels on each side, so the interpolation result of the original reference block size (W) × (H) after interpolation remains the same size (W) × (H). For example, if the original reference block size is 16x16, the interpolation result size is 16x16. Therefore, this type of interpolation process can be referred to as other interpolation (general interpolation). In this embodiment of the present application, other interpolation is referred to as second interpolation, and the blocks in the reference frame that require other interpolation are referred to as second-type blocks.

[0083] It is understood that when saving the interpolation results for each block in the reference frame (the image to be processed), it is necessary to save the first interpolation results corresponding to the first type of blocks and the second interpolation results corresponding to the second type of blocks. For the second interpolation process, such as the other interpolation processes described above, the same image content as the second type of blocks is required to save the second interpolation results, while for the first interpolation process, such as the DMVR interpolation described above, a larger image memory size is required. Therefore, the video processing device can determine the size of the interpolated output image used to save the interpolation results corresponding to each block in the image to be processed based on the size of the image to be processed, thereby determining the required image content size to store the interpolation results corresponding to each block in the image to be processed.

[0084] Specifically, the video processing device can obtain the size of the first type of block and the expanded size of the first type of block. Since the size of the block undergoing DMVR interpolation processing can only be 16×16, 16×8 or 8×16. Therefore, the size of the first type of block is 16×16, 16×8 or 8×16. Among them, the case where the size of the block undergoing DMVR interpolation processing is 16×8 or 8×16 can be understood as the size of the coding block divided by the encoder, and the 16×16 block is further obtained by binary division. The expanded size of the first type of block can include expanding R pixels on each side. The value of R can be determined by the choice of interpolation algorithm and the trade-off between performance and quality. This application does not limit this. For ease of description, this application takes R as 2 as an example, that is, two rows or two columns of pixels are expanded on the top, bottom, left and right respectively.

[0085] Furthermore, the video processing device can determine the size of the output block based on the size and expanded size of the first type of block. The output block is a block in the interpolated output image used to store the first interpolation result obtained by performing DMVR interpolation processing (first interpolation processing) on ​​the first type of block. The size of the output block can be greater than or equal to the size of the first interpolation result. Furthermore, the video processing device can determine the size of the interpolated output image based on the number of blocks in the image to be processed and the size of the output block. Specifically, the size of each block in the image to be processed can be changed to the size of the output block to obtain the size of the interpolated output image.

[0086] Specifically, when the video processing device determines the size of the output block based on the size of the first type of block and the expanded size, the determined output block size may be greater than or equal to the size of the first interpolation result. For example, if the size of the first type of block is 16×16, the expanded size is 2 pixels, and the size of the first interpolation result is 20×20, the video processing device may determine a size greater than 20×20, such as 24×24. Furthermore, the video processing device may determine the size of the interpolated output image based on the number of blocks in the image to be processed and the size of the output block. Specifically, the video processing device may change the size of each block in the image to be processed (e.g., 16×16) to 24×24, thereby calculating the size of the interpolated output block.

[0087] For example, since the smallest side length of the block size (16×16, 16×8 or 8×16) of the first interpolation process (DMVR interpolation process) is 8, the first type of block is 16×16 including 4 8×8 blocks as an example. Figure 6 , Figure 6 This is a schematic diagram of determining the size of an interpolated output image provided by an embodiment of the present application, such as Figure 6 As shown, the image to be processed has a width of w and a height of h. Figure 6 Each 8×8 block shown on the left is expanded by 4 pixels on each side, that is, its size is expanded to 12×12 blocks, and the size of the 16×16 block is expanded to 24×24, resulting in Figure 6 The output block shown on the right is a 24×24 output block consisting of four 12×12 blocks.

[0088] Therefore, for the image to be processed, the size of the interpolated output image can be determined according to the number of blocks and the size of the output block. Specifically, the width and height of the interpolated output image can be determined respectively, thereby obtaining the size of the interpolated output image. Among them, the number of blocks with wide sides in the image to be processed can be determined based on w / 8. Similarly, the number of blocks with high sides in the image to be processed can be determined based on h / 8. Then, according to the size of the blocks in the image to be processed and the size of the output block, the size of the interpolated output image is determined. That is, for a block with a size of 8, the size in the output block is 12. If the above 8×8 block is expanded to 12×12, the width of the interpolated output image is Gao Wei Indicates rounding up.

[0089] It should be noted that the number of blocks in the image to be processed is not the actual number of blocks in the image to be processed. This is because 8 is only the minimum side length in the DMVR interpolation process. By expanding with this as the basic unit, we can cover the above 16×16, 16×8 and 8×16 cases. Therefore, the number of blocks in the image to be processed can be understood as the number of blocks that need to be expanded with this as the basic unit, so that we can Determine the width of the interpolated output image, and based on Determine the height of the interpolated output image and obtain the size of the interpolated output image.

[0090] Furthermore, after determining the size of an image (interpolation output image) for storing interpolation results corresponding to each block in the image to be processed, the video processing device can perform a first interpolation process on each first-type block in the image to be processed to obtain a first interpolation result. Furthermore, a first bidirectional predictive coding process can be performed based on the first interpolation result to determine the MV corresponding to the block to be processed using the first-type block as a reference, and further processing can be performed based on the determined MV, such as determining a prediction block corresponding to the block to be processed.

[0091] S502: Perform a first interpolation process on each first type block in the image to be processed to obtain a first interpolation result.

[0092] In an embodiment of the present application, since DMVR is used to solve the problem of low prediction accuracy of bidirectionally predicted Merge mode coding blocks, the image to be processed is a reference frame of a bidirectionally predicted coding frame (Bi-directional Predicted Frames, B frames). The first type of block in the image to be processed may be a block that requires DMVR interpolation processing. The first interpolation processing may be a DMVR interpolation processing, and the first interpolation result is the interpolation result obtained by performing DMVR interpolation processing on the first type of block. Since the DMVR interpolation processing requires expanding the pixels of each side, for example, expanding each side by 2 pixels, the size of the first interpolation result is larger than the size of the first type of block. In order to better store each first interpolation result, the size of the first interpolation result is less than or equal to the size of the corresponding block in the interpolation output image.

[0093] In one possible implementation, a video processing device performs a first interpolation process on a first type of block in an image to be processed. The first interpolation process may use a preset interpolation algorithm to calculate pixel values ​​in the first type of block to determine expanded pixel values ​​and sub-pixel precision pixel values, such as 1 / 2 pixel (half pixel) pixel values ​​and 1 / 4 pixel pixel values. The preset interpolation algorithm may, for example, include one of the following interpolation algorithms, or a combination of multiple interpolation algorithms: a bilinear interpolation algorithm, a bi-trilinear interpolation algorithm, a 2-tap interpolation filter algorithm, an 8-tap interpolation filter algorithm, and the like. This application does not limit the interpolation algorithm used by the video processing device to perform the first interpolation process. The algorithm may be determined based on factors such as prediction accuracy and encoder performance, and this application does not limit this.

[0094] Therefore, taking the example of expanding each side by 2 pixels, when the size of the first-type block is 16×16, the size of the first interpolation result is 20×20; when the size of the first-type block is 16×8, the size of the first interpolation process is 20×12; and when the size of the first-type block is 8×16, the size of the first interpolation result is 12×20. Furthermore, the video processing device can store the first interpolation results corresponding to each first-type block in an output block in the interpolated output image. It will be understood that the position of the stored output block in the interpolated output image corresponds to the position of the first-type block in the image to be processed. Because the width and height of the output block are respectively larger than those of the first-type block, this can, to a certain extent, avoid overwriting the interpolation results of other sub-blocks when saving the first interpolation result.

[0095] In one possible implementation, the video processing device may perform a second interpolation process on each second type block in the image to be processed to obtain a second interpolation result. The second type block is a block that needs to undergo other interpolation processes. The second interpolation process is other interpolation processes, such as AMVR interpolation processes. The second interpolation result is the interpolation result obtained by performing the second interpolation process on the second type block. For this interpolation process, the size of the interpolation result is the same as the size of the reference block, that is, the size of the second interpolation result is the same as the size of the second type block. The process of the first interpolation process may use a preset interpolation algorithm to calculate the pixel values ​​in the second type block to determine the pixel values ​​with sub-pixel precision. The preset interpolation algorithm may include, for example, one of the following interpolation algorithms, or a combination of multiple interpolation algorithms: a bilinear interpolation algorithm, a bi-trilinear interpolation algorithm, a 2-tap interpolation filter algorithm, an 8-tap interpolation filter algorithm, etc. This application does not limit this.

[0096] It is understood that the image to be processed can serve as a reference frame and can serve as reference blocks for multiple blocks to be processed, wherein some reference blocks are blocks that require DMVR interpolation processing, and some blocks are blocks that require other interpolation processing. Furthermore, the video processing device can perform a first bidirectional predictive coding process based on the first interpolation result to determine the motion vector corresponding to the block to be processed with the first type of block as a reference, and can perform a second bidirectional predictive coding process based on the second interpolation result to determine the motion vector corresponding to the block to be processed with the second type of block as a reference.

[0097] In one possible implementation, when the video processing device is a video encoding device, the video encoding device may, after performing block division, perform predictive encoding on each coding block. When performing inter-frame prediction, the video encoding device may determine the optimal MV for each coding block based on various coding modes. In determining the optimal MV based on various coding modes, the video encoding device may determine that each coding block uses the Merge mode and optimizes the resulting MV using DMVR for subsequent comparison with MVs determined based on other modes. Thus, when determining the MV for each block to be processed (coding block), the video encoding device may determine a first-type block in a reference frame and perform a first interpolation process. It is understood that the video encoding device sequentially determines the MV for each coding block. If the reference frame corresponding to multiple coding blocks is the same frame, the reference frame being the image to be processed. When encoding each coding block, the video encoding device may perform a first interpolation process on the reference block (first-type block) corresponding to the coding block in the image to be processed, obtaining a first interpolation result for each first-type block.

[0098] In another possible implementation, when the video processing device is a video decoding device, the video decoding device can receive a video code stream transmitted by a video encoding device, and then the video decoding device can decode the video code stream to obtain video encoding parameters. The video encoding parameters may include encoding mode information for each block to be processed (decoding block). For example, the Supplemental Enhancement Information (SEI) may include a DMVR usage flag, which can be used to indicate whether a particular video frame uses DMVR. The CU header information may also include a Merge flag, which is used to indicate whether the current CU is encoded using the Merge mode. The video decoding device can then determine the image to be processed corresponding to each decoding block based on the video encoding parameters, and determine each first type block in the image to be processed that requires DMVR interpolation processing. As a result, the video decoding end can simultaneously perform a first interpolation process on each first type block in the image to be processed to obtain a first interpolation result.

[0099] Furthermore, the video decoding device can also determine, based on the video encoding parameters, that other interpolation processing is required for each second-type block in the image to be processed, such as AMVR interpolation processing. As a result, the video decoding end can simultaneously perform a second interpolation processing on each second-type block in the image to be processed to obtain a second interpolation result. Specifically, while the video decoding device is performing the first interpolation processing on each first-type block in the image to be processed, it can simultaneously perform a second interpolation processing on each second-type block in the image to be processed to obtain a first interpolation result and a second interpolation result. It is understandable that the video decoding device can perform the first interpolation processing on each first-type block in the reference frame (the image to be processed) and simultaneously perform the second interpolation processing on each second-type block. This can be understood as extending the block-level interpolation processing to a frame-level interpolation processing. That is, a video frame image (such as the image to be processed) is input, and after interpolation processing is performed, an interpolated output image corresponding to the image to be processed is output. The interpolated output image is a frame-level image.

[0100] For first-type blocks, the size of the first interpolation result corresponding to each first-type block is larger than the size of the first-type block, and the sizes of the first-type blocks are also different, for example, they can be 16×16, 16×8, and 8×16. Therefore, when performing frame-level interpolation processing, the image memory required for other interpolation processing and DMVR interpolation processing is different, and the image memory of the two cannot be reused. If the output block is not expanded, that is, the size of the interpolated output image is the same as the size of the image to be processed, data overlap may occur, that is, the first interpolation result may overlap the surrounding interpolation results, resulting in subsequent incorrect decoding. Therefore, the size of the output block can be expanded. This output block size can be used as the size for storing the first interpolation result or the size for storing the second interpolation result. This can avoid data overlap when performing the first interpolation processing and the second interpolation processing simultaneously and saving the first interpolation result and the second interpolation result.

[0101] For example, this application uses the sizes of the first type block and the second type block as 16×16, 16×8 and 8×16 as examples for explanation. Please refer to Figure 7 , Figure 7 This is a schematic diagram of the principle of storing the interpolation result in the output block provided by the embodiment of the present application. Figure 7 As shown, the leftmost column is the size of the first type block or the second type block, for example, the leftmost size of the first row is 16×16, the leftmost size of the second row is 8×16, and the leftmost size of the third row is 16×8. Figure 7 The middle column shows a schematic diagram of the structure in which the first interpolation result is stored in the output block. For example, the middle of the first row shows that the size of the first interpolation result is 20×20, and the size of the output block is 24×24.

[0102] The middle of the second row shows that if the size of the first type block is 8×16, the two 8×16 blocks are obtained by binary partitioning the 16×16 block. If both blocks are first type blocks, that is, if the first interpolation process is performed on both, the size of the first interpolation result is 12×20, and the size of the output block is 24×24, including two 12×24 sub-blocks, each used to store the first interpolation results corresponding to the two first type blocks. Similarly, the middle of the third row shows that if the size of the first type block is 16×8, the two 16×8 blocks are obtained by binary partitioning the 16×16 block. If both blocks are first type blocks, that is, if the first interpolation process is performed on both, the size of the output first interpolation result is 20×12, and the size of the output block is 24×24, including two 24×12 sub-blocks, each used to store the two first interpolation results.

[0103] in, Figure 7 The rightmost column shows the structure of the second interpolation result stored in the output block. For example, the rightmost column of the first row shows a second interpolation result of 16×16 size, and an output block of 24×24 size. The rightmost column of the second row shows a second-type block of 8×16 size. This is a binary partition of a 16×16 block, resulting in two 8×16 blocks. If both blocks are second-type blocks, i.e., if the second interpolation process is performed on both, the second interpolation result is 8×16 in size, and the output block is 24×24 in size, including two 12×24 sub-blocks, each used to store the two second interpolation results. Similarly, the middle column of the third row shows a first-type block of 16×8 size. This is a binary partition of a 16×16 block, resulting in two 16×8 blocks. If both blocks are second-type blocks, i.e., if the second interpolation process is performed on both, the output second interpolation result is 16×8 in size, and the output block is 24×24 in size, including two 24×12 sub-blocks, each used to store the two second interpolation results.

[0104] It is understandable that Figure 7 Taking the example of dividing a block of size 16×16 into two blocks both of which are first type blocks or both of which are second type blocks, as an example, since the size of the sub-block in the output block is larger than the size of the first interpolation result, such as the width and height of the output block mentioned above are both 4 pixels larger than the width and height of the first type block, and also larger than the size of the second interpolation result, when the 16×16 block is binary divided into two blocks of size 8×16 or two blocks of size 16×8, one of which is a first type block and the other is a second type block, the output blocks can also be saved separately without data overlap, and the first interpolation process and the second interpolation process can also be performed simultaneously.

[0105] It should be noted that Figure 7 When the first interpolation result and the second interpolation result are respectively stored in the output block (a sub-block of the output block), the drawing is performed by taking the storage in the upper left corner area of ​​the output block as an example, and can also be stored in other areas of the output block, which is not limited in this application. It should be understood that after the first interpolation result and the second interpolation result are stored in the output block in the interpolation output image, when the video decoding device performs decoding processing on the decoded block (current block), the first interpolation result or the second interpolation result corresponding to the current block can be extracted from the interpolation output image to perform bidirectional predictive coding processing (such as the first bidirectional predictive coding processing or the second bidirectional predictive coding processing) to determine the optimal MV of the current block.

[0106] In the process of extracting the first interpolation result or the second interpolation result corresponding to the current block from the interpolated output image, the video decoding device can specifically determine the output block at the corresponding position in the interpolated output image as the first interpolation result or the second interpolation result corresponding to the current block based on the position of the reference block corresponding to the current block in the reference frame (the image to be processed). In some embodiments, when the size of the reference block is 16×8 or 8×16, since only an output block including two sub-blocks can be determined when determining the output block at the corresponding position, it is also necessary to specifically determine which of the two sub-blocks it is. Specifically, the determination can be made based on the positional relationship between the two blocks after binary division and the positional relationship between the two sub-blocks included in the output block. The two blocks after binary division can be two first-type blocks, two second-type blocks, or one first-type block and one second-type block, respectively.

[0107] In one possible implementation, a video processing device includes a graphics processing unit (GPU) and a central processing unit (CPU). If the video processing device is a video decoding device, the video decoding device may include a decoder that combines CPU and GPU heterogeneous processing to decode video streams. Because the GPU has high computing power (parallel computing capability), it can run a large number of calculations in parallel. This can be leveraged to execute more complex algorithms, while the CPU only needs to input data to the GPU and collect calculation results from the GPU. In VVC, interpolation processing for inter-frame prediction is the most time-consuming step in the encoding and decoding process. By porting this processing to the GPU, the time consumed may be significantly less than running it on the CPU. For video encoding devices, the GPU can replace the CPU in performing interpolation operations for inter-frame prediction, which is faster and more efficient than performing interpolation operations on the CPU. For video decoding devices, the GPU can simultaneously perform multiple types of interpolation on images, which can improve decoding speed and fully utilize the GPU's computing power. This can save the overall time of video encoding and decoding, increase the overall speed of the encoder and decoder, and improve the efficiency of video encoding and decoding.

[0108] The video processing device can call the GPU for interpolation processing through an application programming interface (API), which can be Open Computing Language (OpenCL), Open Graphics Library (OpenGL), Vulkan Graphics and Compute API (Vulkan), or Metal Graphics and Compute API (Metal). The API can provide a standardized interface for accessing and controlling GPU functions, that is, for interacting with and controlling the GPU, allowing developers to write applications that can be efficiently executed on the GPU, thereby utilizing the powerful computing power of the GPU to perform high-performance tasks such as graphics rendering and parallel computing. For example, the task of interpolating the image to be processed in the inter-frame prediction mentioned above can be applied to video encoding and decoding of various different standards, and can also be applied to encoders and decoders containing multiple types of interpolation processing algorithms, such as the AS3 decoder containing the DMVR algorithm.

[0109] Furthermore, the video processing device can perform a first bidirectional predictive coding process based on the first interpolation result to determine the motion vector corresponding to the block to be processed with the first type of block as a reference, and the video processing device can also perform a second bidirectional predictive coding process based on the second interpolation result to determine the motion vector corresponding to the block to be processed with the second type of block as a reference.

[0110] S503 : Perform a first bidirectional predictive coding process according to the first interpolation result to determine a motion vector corresponding to the block to be processed with the first type block as a reference.

[0111] In the embodiment of the present application, since the result of the interpolation process can be used for sub-pixel precision search processing, that is, searching around the existing MV to obtain a better MV, the first bidirectional predictive coding process can refer to the video processing device determining the motion vector of the block to be processed based on the first interpolation result with the first type of block as a reference, that is, further optimizing the existing MV. The block to be processed can be understood as an encoding block for the video encoding device and as a decoding block for the video decoding device.

[0112] In one possible implementation, a video processing device may obtain a motion vector of a current block to be processed at integer pixel precision. The video processing device may process each block to be processed in sequence. The currently processed block may be referred to as the current block, i.e., the current coding unit (CU) or the current decoding unit. The MV at integer pixel precision means that the accuracy of the MV is at the integer pixel level. Specifically, the video processing device may obtain the position information of the current block and the initial MV corresponding to the current block, and then determine the target first type corresponding to the current block in the image to be processed based on the position information and the initial MV of the current block. Finally, integer pixel search processing is performed based on the target first type block to obtain the MV of the current block at integer pixel precision.

[0113] Among them, the position information of the current block can be represented by the coordinates of the upper left corner of the current block, and the initial motion vector corresponding to the current block can be the MV before DMVR optimization, which can be understood as the MV determined in Merge mode. In the case where the video processing device is a video encoding device, the initial MV can be an optimal candidate MV selected from a set candidate MV list. The set candidate MV list can be a Merge candidate MV list, which can be constructed for the video encoding device, and an optimal candidate MV can be selected therefrom, and then the MV index of the candidate MV can be encoded into the bitstream and transmitted to the video decoding device. In the case where the video processing device is a video decoding device, the video decoding device can decode the video bitstream to obtain video encoding parameters, which include the motion vector index of the current block. According to the MV index, the candidate MV indicated by the MV index is determined from the set candidate MV list as the initial MV corresponding to the current block.

[0114] Among them, the set candidate MV list can be a candidate MV list constructed by the video decoding device in the same way as the video encoding device. Specifically, the video processing device can construct a Merge MVP candidate list, and then obtain a candidate MV list based on MV=MVP. The Merge mode includes a total of 5 candidate MVP types. The VVC standard defines the order in which they construct the Merge list, which are, from first to last, spatial candidates, temporal candidates, history-based candidates (History-Based Motion Vector Prediction, HMVP), pairwise average (Pair-wise) candidates, and zero values. Please refer to Figure 8 , Figure 8 This is a schematic diagram of the principle of constructing and determining candidate MVPs. Specifically, it can be determined based on the MV of the coding block (decoding block) associated with the current block. Figure 8 As shown in (a), the Merge mode contains 5 spatial candidate MVPs, and the Merge list can be constructed in the order of {A1, A0, B2, B0, B1}, where A1, A0, B2, B0, and B1 can be blocks adjacent to the current block in the spatial domain, and the spatial candidate MVP can be determined based on the MV of the blocks adjacent to the current block in the spatial domain.

[0115] Furthermore, the Merge mode includes two time domain candidate MVPs, namely Figure 8{D1, D0} shown in (b) of the figure. Temporal MVP candidates can be determined based on the MVs of blocks co-located with the current block but not in the same frame. The VVC standard also allows for customizing the length of the Merge list. During the Merge list construction process, if the Merge list is full, construction stops. HMVP involves constructing and maintaining a queue of size 5 to record the motion information of previous blocks. When encoding or decoding the current block, HMVPs are added to the Merge list in order from end to beginning until the list is full. HMVP technology uses a first-in-first-out (FIFO) queue to store motion information of previous blocks. If the stored prediction candidate information is the same as the motion information just encoded, the duplicate candidate information is first removed. Then all HMVP candidates are moved forward, and the motion information of the current block is added to the end of the FIFO queue. If the motion information of the current block is different from the motion information of any candidate in the FIFO queue, the latest motion information is added to the end of the FIFO queue. When adding new motion information to the HMVP list, if the list has reached the maximum length, the first candidate information in the FIFO queue is removed, and the latest motion information is added to the end of the FIFO queue.

[0116] Furthermore, the pairwise average candidate is generated by averaging the first two merge candidates in the existing merge candidate list. These first two candidates can be, for example, the spatial candidate and temporal candidate mentioned above. A zero-valued candidate refers to a MV with a value of (0, 0) that can be added to the end of the merge candidate list if there are still empty positions in the merge candidate list for the current block, bringing the length of the merge MVP candidate list to the specified number. Thus, the video processing device constructs the specified candidate MV list.

[0117] In one possible implementation, the video processing device can obtain the initial MV corresponding to the current block from the Merge candidate list (the set MV candidate list). For the video encoding device, the best candidate MV can be selected from the set MV candidate list based on RDO as the initial MV corresponding to the current block, and then the corresponding MV index can be transmitted to the video decoding device. For example, the video encoding device can traverse the set MV candidate list, calculate their respective RDOs, and then select the candidate MV with the smallest RDO as the best candidate MV. Furthermore, the video processing device can determine the target first type block corresponding to the current block in the image to be processed based on the position information and the initial motion vector of the current block, that is, the reference frame corresponding to the current block in the image to be processed. Then, based on the position information of the current block at the same position in the reference frame and the initial MV, the reference block corresponding to the current block in the reference frame, that is, the target first type block, can be determined.

[0118] Furthermore, the video processing device can perform an integer-pixel search based on the target first-type block to obtain the MV of the current block at integer-pixel precision, that is, to obtain the optimized MV at integer-pixel precision. The integer-pixel search can be performed within a search range of two integer pixels near the initial MV, for a total of 25 integer pixels. For example, the video processing device can perform an exhaustive search within the search range, i.e., traversing each pixel within the search range and selecting the one with the smallest Sum of Absolute Differences (SAD) to obtain the MV at integer-pixel precision. For another example, to reduce search complexity, an early termination algorithm, such as a 2-iteration search scheme, can be used. This algorithm only compares the prediction errors (SAD values) of the center point (the point where the initial MV is located) and its four surrounding points. If the center point has the smallest SAD, the integer-pixel search phase terminates early. Otherwise, the search continues for the point with the smallest SAD as the new center point for a second round of search, until the integer-pixel search phase terminates.

[0119] In one possible implementation, after the video processing device obtains the MV of the current block at integer pixel precision, it can extract a first interpolation result for the target first-type block from the interpolated output image. Specifically, based on the position of the reference block (target first-type block) corresponding to the current block in the reference frame (the image to be processed), the output block at the corresponding position in the interpolated output image is determined to obtain the first interpolation result for the target first-type block. This output block is the block in the interpolated output image used to store the first interpolation result for the target first-type block. The video processing device can then perform a first sub-pixel search based on the first interpolation result to obtain a first MV offset corresponding to the current block.

[0120] Specifically, the video processing device can use the MV at integer pixel precision as the center point, determine the sub-pixel positions of the center point and the surrounding sub-pixel positions, such as at intervals of 1 / 4 pixel or 1 / 8 pixel, calculate the SAD of each sub-pixel position, and then determine the sub-pixel position with the smallest SAD as the optimal sub-pixel MV offset, that is, the first MV offset, and the first MV is the offset value at sub-pixel precision. Similarly, in order to reduce complexity, the first sub-pixel search process can use a parametric error surface equation to solve the optimal sub-pixel MV offset to obtain the first MV offset. Furthermore, the video processing device can correct the MV of the current block at integer pixel precision according to the first MV offset to obtain the first corrected MV corresponding to the current block. That is, the first corrected MV can be understood as the MV at integer pixel precision + the first MV offset value.

[0121] In one possible implementation, when the video processing device is a video encoding device, the video encoding device can respectively use various encoding modes to determine the MV, and finally select the optimal MV based on RDO or SAD. The above process is the MV determined by the video encoding device in the Merge mode and optimized by DMVR, that is, the first corrected MV. The video encoding device can also use other encoding modes to determine the MV and then select the optimal MV. Specifically, the video encoding device can perform a second interpolation process on the target first type block to obtain a second interpolation result of the target first type block, wherein the size of the second interpolation result is the same as the size of the target first type block. That is, the video encoding device can perform other interpolation processes on the reference block (target first type block) corresponding to the current block.

[0122] Specifically, the video encoding device can perform a second pixel search process based on the second interpolation result of the target first type block to obtain a second MV offset corresponding to the current block, and then correct the MV of the current block at integer pixel precision according to the second MV offset to obtain the second corrected MV corresponding to the current block. Finally, the optimal MV corresponding to the current block can be determined based on the first corrected MV and the second corrected MV. It should be noted that the target first type block here is explained by taking the example of the video encoding device determining that the reference blocks in different encoding modes are all target first type blocks. In the actual encoding process, there may also be a situation where the initial MV determined by the video encoding device is different, resulting in different reference blocks corresponding to the current block in different encoding modes. In the embodiment of the present application, the MV of the current block at integer pixel precision is the same in different encoding modes, and different algorithms, such as DMVR and AMVR, can be used to correct the MV at integer pixel precision for explanation.

[0123] The process types of the second sub-pixel search process and the first sub-pixel search process are explained using the example of the second interpolation process being an AMVR interpolation process. The sub-pixel precision of the second sub-pixel search process in AMVR may be different. For example, the precision of the first sub-pixel search process is 1 / 8, and the precision of the second sub-pixel search process is 1 / 2, thereby obtaining the optimal sub-pixel MV offset, i.e., the second MV offset. Furthermore, the video encoding device can determine the second corrected MV based on the motion vector at integer pixel precision + the second MV offset value. Thus, the video encoding device can determine the optimal MV corresponding to the current block based on the first corrected MV and the second corrected MV. Specifically, the video encoding device can select an MV that meets a preset error condition from the first corrected MV and the second corrected MV, and use the selected MV as the optimal MV corresponding to the current block.

[0124] In one possible implementation, the video encoding device may obtain a preset RDO threshold, and then determine the RDOs corresponding to the first and second revised MVs, respectively. The MV with an RDO less than the preset RDO threshold is determined as the MV that satisfies the preset error condition and serves as the optimal MV for the current block. In another possible implementation, the video encoding device may also obtain a preset SAD threshold, and then determine the SADs corresponding to the first and second revised MVs, respectively. The MV with an SAD less than the preset SAD threshold is determined as the MV that satisfies the preset error condition and serves as the optimal MV for the current block. In yet another possible implementation, the video encoding device may determine the RDOs (or SADs) corresponding to the first and second revised MVs, respectively, and determine the MV corresponding to the smaller RDO (or SAD) as the MV that satisfies the preset error condition and serves as the optimal MV for the current block.

[0125] In an embodiment of the present application, when the video encoding device determines that the first corrected MV is the optimal MV corresponding to the current block, the video encoding device can encode the MV index indicating the position of the MV in the Merge candidate MV into the video encoding parameters and transmit it to the video decoding device. Among them, the VVC standard defines that DMVR is only applicable to bidirectional prediction, then the above-mentioned first corrected motion vector includes a forward motion vector and a backward motion vector, and the interpolated output image includes a forward reference frame and a backward reference frame. It can be understood that in the bidirectional Merge mode, the Merge candidate MVs are multiple groups of MV pairs, and each MV in the MV pair (such as MV0 and MV1) is associated with a reference frame, and the image to be processed can be the reference frame associated with each MV in the MV pair. The video processing device can perform integer pixel search processing and first sub-pixel search processing on the Merge candidate MV pairs respectively to determine the forward motion vector and the backward motion vector.

[0126] In one possible implementation, the video processing device may determine a first prediction block corresponding to the current block in a forward reference frame based on the position information and forward motion vector of the current block; and determine a second prediction block corresponding to the current block in a backward reference frame based on the position information and backward motion vector of the current block. The video processing device may extract a block of the same size as the current block in the reference frame as a prediction block based on the first corrected MV. Specifically, the first prediction block may be determined in the forward reference frame based on the position information of the current block, the same block in the same position as the current block, and the forward motion vector. Similarly, the second prediction block may be determined in the backward reference frame based on the position information of the current block, the same block in the same position as the current block, and the backward motion vector.

[0127] Furthermore, the video processing device may perform weighted prediction processing based on the first prediction block and the second prediction block to obtain a prediction block corresponding to the current block. The video processing device may respectively obtain weights corresponding to the first prediction block and the second prediction block, and then perform weighted processing on the first prediction block and the second prediction block to obtain a prediction block corresponding to the current block. For example, when the weight is 1, the pixel value of the prediction block corresponding to the current block is the average of the pixel values ​​of each pixel point in the first prediction block and the second prediction block. Thus, by searching for a more accurate MV, a prediction block of better quality may be obtained.

[0128] If the video processing device is a video encoding device, the video encoding device can perform a subtraction between the current block and the predicted block to obtain residual data. The residual data can then be transformed, quantized, entropy encoded, etc., and the residual data can be added to the video encoding data for transmission to the video decoding device. If the video processing device is a video decoding device, the video decoding device can decode the video stream to obtain video encoding data, which can include the residual data of the current block. Based on the determined prediction block and residual data, the video decoding device can combine the two to obtain a reconstructed block. After the video decoding device obtains the reconstructed block corresponding to each current block (decoded block), it obtains the video content transmitted by the video encoding device.

[0129] The following describes the processing flows of the video encoding device and the video decoding device respectively:

[0130] (1) Video encoding equipment

[0131] When encoding a video to be encoded, a video encoding device can divide each video frame image into multiple non-overlapping processing units, which can be CUs or coding blocks. The coding block currently being processed is the current block. Taking the current block in a B-frame as an example, the video encoding unit can determine the MV and prediction block of the current block based on various inter-frame prediction coding modes, and then select the optimal MV as the MV of the current block for transmission to the video decoding device. The video encoding device can first construct a set of Merge candidate MVs, then select an optimal candidate MV from the Merge candidate MVs based on RDO. It can then perform DMVR on the selected optimal candidate MV, such as the first interpolation and first bidirectional predictive coding processes described above, to obtain a first modified MV. The video encoding device can also use other methods to determine other modified MVs, such as the second interpolation and second interpolation predictive coding processes described above, to obtain a second modified MV.

[0132] During interpolation, the video encoding device can determine a larger output block size and an interpolated output image size larger than the reference frame to store the interpolation results of the reference block in the reference frame, thereby improving encoding accuracy. Furthermore, the video encoding device can perform interpolation using a GPU to improve processing speed. Furthermore, the video encoding device can determine the optimal MV corresponding to the current block from the first and second modified MVs. After determining the optimal MV, which includes a forward MV and a backward MV, a prediction block can be determined in the forward reference frame and the backward reference frame based on the forward MV, and then weighted prediction can be performed to obtain the prediction block corresponding to the current block. The video encoding device can encode coding mode information (video coding parameters) and residual data (video encoding data) into a video stream and transmit it to a video decoding device. The coding mode information can include, for example, a merge mode flag, a DMVR usage flag, an AMVR usage flag, and an MV index for the current block. The residual data is the difference between the current block and the prediction block and can be further processed through transformation, quantization, and entropy coding to obtain the video encoding data.

[0133] (2) Video decoding equipment

[0134] A video decoding device can receive a video stream of a video to be decoded and decode the stream to obtain video coding parameters and video coding data. The video decoding device can then perform entropy decoding, inverse quantization, and inverse transformation on the video coding data to obtain residual data. The video decoding device then obtains the video coding parameters and, based on the video coding parameters, determines which blocks in a reference frame (the image to be processed) are of the first type and which blocks are of the second type. This determination can be made based on the coding mode information in the video coding parameters. After determining the first and second type blocks in the image to be processed, the device can interpolate each block in the image to be processed and save the interpolation results. The video decoding device can determine a larger output block size and a larger interpolated output image size than the reference frame to store the interpolation results of the reference blocks in the reference frame, thereby improving decoding accuracy. Furthermore, the video encoding device can utilize a GPU to perform frame-level interpolation to increase decoding speed.

[0135] Furthermore, when decoding a decoded block in a B-frame, the currently processed decoded block can be referred to as the current block. The video decoding device can determine the initial MV of the current block based on the MV index, and then perform the first or second bidirectional predictive coding process. The MV index can, for example, be used to determine a candidate MV in a Merge candidate MV, and then perform a DMVR operation, such as extracting the first interpolation process corresponding to the current block from the interpolated output image to perform the first bidirectional predictive coding process and obtain a first revised MV. If, based on coding mode information, it is determined that the current block uses another mode to optimize the MV, the video decoding device can extract the first interpolation process corresponding to the current block from the interpolated output image to perform the second bidirectional predictive coding process and obtain a revised MV corresponding to the current block. Furthermore, the video decoding device can separately determine the forward MV and backward MV included in the revised MV (such as the first revised MV). Based on the forward MV, a prediction block can be determined in each of the forward and backward reference frames, and then weighted prediction can be performed to obtain the prediction block corresponding to the current block.

[0136] Finally, the predicted block corresponding to the current block is combined with the residual data to obtain a reconstructed block corresponding to the current block. Optionally, this reconstructed block can also be subjected to loop filtering to improve its quality and reduce distortion and blocking artifacts introduced during the encoding process. After determining the corresponding reconstructed block for each block to be processed in a B-frame, the B-frame image can be reconstructed. Thus, the video decoding device can reconstruct each video frame in the video to be decoded, thereby obtaining the video content corresponding to the video to be decoded.

[0137] In the technical solutions provided in some embodiments of the present application, after determining the size of an interpolated output image larger than the size of the image to be processed based on the size of the image to be processed, a first interpolation process can be performed on each first-type block in the image to be processed to obtain a first interpolation result, wherein the size of the first interpolation result is larger than the size of the first-type block and equal to or greater than the size of the corresponding block in the interpolated output image. A first bidirectional predictive coding process can then be performed based on the first interpolation result to determine the motion vector corresponding to the block to be processed with the first-type block as a reference. Thus, by determining a larger interpolated output image size, the size of each block in the interpolated output image can be larger than the size of the first interpolation result corresponding to each first-type block. When storing the first interpolation result larger than the size of the first-type block, the situation of overlapping the interpolation results of other sub-blocks can be avoided to a certain extent, thereby improving the accuracy of encoding and decoding.

[0138] The above content introduces the specific execution process of the method for determining a motion vector and a system for determining a motion vector suitable for implementing the method for determining a motion vector. The following describes the scenarios in which the method for determining a motion vector is applicable:

[0139] (1) Live video streaming scenario

[0140] Please also refer to Figure 9 , Figure 9 FIG. 1 is a schematic diagram of an application of a method for determining a motion vector provided by an embodiment of the present application in a live video broadcasting scenario. Figure 9 As shown, in a live video broadcast scenario, the video encoding device can be an electronic device used by the anchor user. The video encoding device can provide video production and video encoding functions, and a live video broadcast application can be run on the video encoding device. The video decoding device is an electronic device used by the audience user who watches the anchor's live broadcast. The video decoding device can provide the function of decoding the video stream and then rendering and playing the obtained video content. The video encoding device and the video decoding device can be devices including a CPU and a GPU for performing video encoding and decoding processing.

[0141] Among them, the video encoding device can obtain video content and perform video encoding processing to obtain a video code stream. The video encoding device can use the push streaming tool to push the video package file containing the video code stream to the cloud server through the RTMP transmission protocol. After the cloud server receives the video package file, the cloud server can decode the video package file based on user needs, and then perform processing such as transcoding, denoising, enhancement, and analysis. Among them, the video encoding device can also use the push streaming tool to push the video package file to the content delivery network (Content Delivery Network, CDN) through the RTMP transmission protocol. The cloud server can pull the video package file from the CDN and process the video package file based on user needs. Then, the video decoding device of the viewer user can pull the video package file through the video live broadcast application, unseal and decode the video package file, and render the decoded video content based on display parameters, and then display (play) the video content of the video code stream.

[0142] In live video streaming scenarios, video encoding and decoding devices can use GPUs to perform interpolation processing during the video encoding and decoding process. Leveraging the GPU's greater parallel computing capabilities, this can increase encoding and decoding speeds, saving encoding and decoding time and improving real-time data transmission. Furthermore, when saving interpolation results, a larger interpolated output image can be used to store the results. This can, to a certain extent, avoid the interpolation results of other sub-blocks, thereby improving encoding and decoding accuracy.

[0143] (2) Video on demand scenario

[0144] Video on Demand (VOD) refers to the ability to play corresponding video content according to the requirements of the viewer. It can also be understood as transmitting the content that the user wants to watch (click or select) to the requesting user. Please also refer to Figure 10 , Figure 10 FIG. 1 is a schematic diagram of an application of a video stream processing method provided in an embodiment of the present application in a video on demand scenario. Figure 10 In a video-on-demand scenario, the video encoding device can be the electronic device of the platform or user that provides video content. This video encoding device can provide video production, video encoding, and video packaging functions. The video decoding device is the electronic device used by viewers who use the video-on-demand function. The video decoding device can provide the function of decoding the video stream and then rendering and playing the video. The video encoding device and video decoding device can be devices including a CPU and a GPU to perform video encoding and decoding.

[0145] Specifically, the video encoding device can encode the video content to obtain a video stream. The video encoding device can then store the video encapsulation file containing the video stream in cloud storage, such as a cloud server. Furthermore, the video decoding device can retrieve the corresponding video encapsulation file from the cloud storage based on the video content selected by the user, decapsulate and decode the video encapsulation file, and render the decoded video content based on display parameters to display (play) the video content of the video stream.

[0146] In video-on-demand scenarios, video encoding and decoding devices can also use the GPU to perform interpolation processing during the video encoding and decoding process, fully utilizing the GPU's computing power and improving encoding and decoding efficiency. When saving the interpolation results, a larger interpolated output image can be used to store the interpolation results. This can, to a certain extent, avoid the interpolation results of other sub-blocks, thereby improving the accuracy of encoding and decoding.

[0147] The above describes in detail the method of the embodiment of the present application. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0148] See also Figure 11 , Figure 11 This is a schematic diagram of the structure of a motion vector determination device provided in an embodiment of the present application. The motion vector determination device 110 can be used to perform Figure 5 The corresponding steps in the method for determining the motion vector shown in FIG. The motion vector determination device 110 includes the following units:

[0149] A determining unit 1101 is configured to determine a size of an interpolated output image corresponding to the image to be processed according to a size of the image to be processed, wherein the size of the interpolated output image is larger than the size of the image to be processed;

[0150] The processing unit 1102 is configured to perform a first interpolation process on each first-type block in the image to be processed to obtain a first interpolation result, where the size of the first interpolation result is larger than the size of the first-type block and the size of the first interpolation result is smaller than or equal to the size of the corresponding block in the interpolated output image; and perform a first bidirectional predictive coding process based on the first interpolation result to determine a motion vector corresponding to the block to be processed with the first-type block as a reference.

[0151] In a possible implementation, the determining unit 1101 is configured to determine the size of the interpolated output image corresponding to the image to be processed according to the size of the image to be processed, specifically:

[0152] Acquire the size of the first type block and the expanded size of the first type block;

[0153] determining a size of an output block according to a size of the first type of block and the expanded size;

[0154] The size of the interpolated output image is determined according to the number of blocks in the image to be processed and the size of the output block.

[0155] In one possible implementation, the processing unit 1102 is configured to perform a first bidirectional predictive coding process based on the first interpolation result to determine a motion vector corresponding to a block to be processed with the first type of block as a reference, specifically:

[0156] Obtaining the motion vector of the current block to be processed at integer pixel precision;

[0157] extracting a first interpolation result of a target first-type block from the interpolation output image, where the target first-type block is a reference block of the current block;

[0158] performing a first sub-pixel search process based on the first interpolation result to obtain a first motion vector offset corresponding to the current block;

[0159] The motion vector of the current block at integer pixel precision is corrected according to the first motion vector offset to obtain a first corrected motion vector corresponding to the current block.

[0160] In a possible implementation, the processing unit 1102 is configured to obtain a motion vector of a current block to be processed at integer pixel precision, specifically:

[0161] Obtaining position information of the current block and an initial motion vector corresponding to the current block;

[0162] determining, in the image to be processed, the target first-type block corresponding to the current block according to the position information of the current block and the initial motion vector;

[0163] An integer pixel search process is performed on the target first-type block to obtain a motion vector of the current block at integer pixel accuracy.

[0164] In a possible implementation, the first corrected motion vector includes a forward motion vector and a backward motion vector, the interpolated output image includes the forward reference frame and the backward reference frame, and the determining unit 1101 is further configured to determine, in the forward reference frame, a first prediction block corresponding to the current block based on the position information of the current block and the forward motion vector;

[0165] Determining, in the backward reference frame, a second prediction block corresponding to the current block according to the position information of the current block and the backward motion vector;

[0166] A weighted prediction process is performed on the first prediction block and the second prediction block to obtain a prediction block corresponding to the current block.

[0167] In a possible implementation, the processing unit 1102 is configured to obtain the initial motion vector corresponding to the current block, specifically to:

[0168] Decoding the video code stream to obtain video coding parameters, wherein the video coding parameters include a motion vector index of the current block;

[0169] According to the motion vector index, a candidate motion vector indicated by the motion vector index is determined from a set candidate motion vector list to serve as an initial motion vector corresponding to the current block.

[0170] In one possible implementation, the processing unit 1102 is further configured to perform a second interpolation process on the target first-type block to obtain a second interpolation result of the target first-type block, where the size of the second interpolation result of the target first-type block is the same as the size of the target first-type block; perform a second sub-pixel search process based on the second interpolation result of the target first-type block to obtain a second motion vector offset corresponding to the current block; and correct the motion vector of the current block at integer pixel precision according to the second motion vector offset to obtain a second corrected motion vector corresponding to the current block.

[0171] The determining unit 1101 is further configured to determine an optimal motion vector corresponding to the current block according to the first corrected motion vector and the second corrected motion vector.

[0172] In a possible implementation, the determining unit 1101 is configured to determine the optimal motion vector corresponding to the current block according to the first corrected motion vector and the second corrected motion vector, specifically:

[0173] A motion vector that meets a preset error condition is selected from the first corrected motion vector and the second corrected motion vector, and the selected motion vector is used as the optimal motion vector corresponding to the current block.

[0174] In one possible implementation, the processing unit 1102 is further configured to perform a second interpolation process on each second-type block in the image to be processed to obtain a second interpolation result, where the size of the second interpolation result is the same as the size of the second-type block; and perform a second bidirectional predictive coding process based on the second interpolation result to determine a motion vector corresponding to the block to be processed with the second-type block as a reference.

[0175] In a possible implementation, the processing unit 1102 is further configured to, while performing the first interpolation processing on each first type block in the image to be processed to obtain the first interpolation result, simultaneously perform the second interpolation processing on each second type block in the image to be processed to obtain the second interpolation result.

[0176] In a possible implementation, the first interpolation processing is performed on each of the first type blocks in the image to be processed, and the second interpolation processing is performed on each of the second type blocks in the image to be processed, using a graphics processor (GPU) of a video processing device.

[0177] According to one embodiment of the present application, Figure 5 The steps involved in the method shown can all be performed by Figure 11The motion vector determination device shown is executed by various units. For example, Figure 5 Step S501 shown is performed by Figure 11 The determination unit 1101 shown is executed, and steps S502 and S503 are performed by Figure 11 The processing unit 1102 is shown to execute.

[0178] According to one embodiment of the present application, Figure 11 The various units in the motion vector determination device 110 shown can be individually or all combined into one or several other units to form a whole, or one (or some) of the units can be further divided into multiple functionally smaller units to form a whole, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the motion vector determination device 110 can also include other units. In actual applications, these functions can also be implemented with the assistance of other units and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, the following can be executed by running on a general-purpose computing device of a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 5 A computer program (including program code) for each step involved in the corresponding method shown in Figure 11 The computer program can be recorded on a computer-readable storage medium, for example, and loaded into a computer-readable storage medium. Figure 1 The motion vector determination system shown is located in a video processing device and operates therein.

[0179] Based on the description of the above-mentioned method for determining a motion vector, the present invention also discloses a device for determining a motion vector. Figure 12 The motion vector determination device 120 may include at least a processor 1201, an input device 1202, an output device 1203, and a memory 1204. The processor 1201, the input device 1202, the output device 1203, and the memory 1204 in the motion vector determination device 120 may be connected via a bus or other means.

[0180] The above-mentioned memory 1204 is a memory device in the motion vector determination device 120, which is used to store programs and data. It can be understood that the memory 1204 here can include both the built-in storage medium of the motion vector determination device and, of course, the extended storage medium supported by the motion vector determination device 120. The memory 1204 provides a storage space, which stores the operating system of the motion vector determination device 120. In addition, the computer program (including program code) is also stored in the storage space. It should be noted that the computer storage medium here can be a high-speed RAM memory; optionally, it can also be at least one computer storage medium away from the aforementioned processor. The aforementioned processor can be called a central processing unit (CPU), which is the core and control center of the motion vector determination device and is used to run the computer program stored in the above-mentioned memory 1204.

[0181] In one implementation, the processor 1201 may load and execute a computer program stored in the memory 1204 to implement corresponding steps of the method in the above embodiment of the method for determining a motion vector. Specifically, the processor 1201 loads and executes the computer program stored in the memory 1204 to:

[0182] determining the size of an interpolation output image corresponding to the image to be processed according to the size of the image to be processed, wherein the size of the interpolation output image is larger than the size of the image to be processed;

[0183] performing a first interpolation process on each first-type block in the image to be processed to obtain a first interpolation result, where the size of the first interpolation result is larger than the size of the first-type block and the size of the first interpolation result is smaller than or equal to the size of the corresponding block in the interpolation output image;

[0184] A first bidirectional predictive coding process is performed according to the first interpolation result to determine a motion vector corresponding to a block to be processed with the first type of block as a reference.

[0185] In one possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204 for determining, according to the size of the image to be processed, the size of the interpolated output image corresponding to the image to be processed, specifically for:

[0186] Acquire the size of the first type block and the expanded size of the first type block;

[0187] determining a size of an output block according to a size of the first type of block and the expanded size;

[0188] The size of the interpolated output image is determined according to the number of blocks in the image to be processed and the size of the output block.

[0189] In one possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204 for performing a first bidirectional predictive coding process based on the first interpolation result to determine a motion vector corresponding to a block to be processed with the first type of block as a reference, specifically for:

[0190] Obtaining the motion vector of the current block to be processed at integer pixel precision;

[0191] extracting a first interpolation result of a target first-type block from the interpolation output image, where the target first-type block is a reference block of the current block;

[0192] performing a first sub-pixel search process based on the first interpolation result to obtain a first motion vector offset corresponding to the current block;

[0193] The motion vector of the current block at integer pixel precision is corrected according to the first motion vector offset to obtain a first corrected motion vector corresponding to the current block.

[0194] In one possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204 for obtaining a motion vector of a current block to be processed at integer pixel precision, specifically for:

[0195] Obtaining position information of the current block and an initial motion vector corresponding to the current block;

[0196] determining, in the image to be processed, the target first-type block corresponding to the current block according to the position information of the current block and the initial motion vector;

[0197] An integer pixel search process is performed on the target first-type block to obtain a motion vector of the current block at integer pixel accuracy.

[0198] In a possible implementation, the first corrected motion vector includes a forward motion vector and a backward motion vector, the interpolated output image includes the forward reference frame and the backward reference frame, and the processor 1201 loads and executes a computer program stored in the memory 1204, and is further configured to:

[0199] Determining, in the forward reference frame, a first prediction block corresponding to the current block according to the position information of the current block and the forward motion vector;

[0200] Determining, in the backward reference frame, a second prediction block corresponding to the current block according to the position information of the current block and the backward motion vector;

[0201] A weighted prediction process is performed on the first prediction block and the second prediction block to obtain a prediction block corresponding to the current block.

[0202] In a possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204 for obtaining an initial motion vector corresponding to the current block, specifically for:

[0203] Decoding the video code stream to obtain video coding parameters, wherein the video coding parameters include a motion vector index of the current block;

[0204] According to the motion vector index, a candidate motion vector indicated by the motion vector index is determined from a set candidate motion vector list to serve as an initial motion vector corresponding to the current block.

[0205] In one possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204, and is further configured to:

[0206] performing a second interpolation process on the target first-type block to obtain a second interpolation result of the target first-type block, wherein a size of the second interpolation result of the target first-type block is the same as a size of the target first-type block;

[0207] Performing a second sub-pixel search process based on a second interpolation result of the target first-type block to obtain a second motion vector offset corresponding to the current block;

[0208] Correcting the motion vector of the current block at integer pixel precision according to the second motion vector offset to obtain a second corrected motion vector corresponding to the current block;

[0209] An optimal motion vector corresponding to the current block is determined according to the first corrected motion vector and the second corrected motion vector.

[0210] In one possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204 for determining, based on the first corrected motion vector and the second corrected motion vector, an optimal motion vector corresponding to the current block, specifically for:

[0211] A motion vector that meets a preset error condition is selected from the first corrected motion vector and the second corrected motion vector, and the selected motion vector is used as the optimal motion vector corresponding to the current block.

[0212] In one possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204, and is further configured to:

[0213] performing a second interpolation process on each second-type block in the image to be processed to obtain a second interpolation result, where the size of the second interpolation result is the same as the size of the second-type block;

[0214] A second bidirectional predictive coding process is performed according to the second interpolation result to determine a motion vector corresponding to a block to be processed with the second type block as a reference.

[0215] In one possible implementation, the processor 1201 loads and executes a computer program stored in the memory 1204 for performing the second interpolation processing on each first type block in the image to be processed to obtain the first interpolation result while simultaneously performing the second interpolation processing on each second type block in the image to be processed to obtain the second interpolation result.

[0216] In a possible implementation, the first interpolation processing is performed on each of the first type blocks in the image to be processed, and the second interpolation processing is performed on each of the second type blocks in the image to be processed, using a graphics processor (GPU) of a video processing device.

[0217] In the technical solutions provided in some embodiments of the present application, after determining the size of an interpolated output image larger than the size of the image to be processed based on the size of the image to be processed, a first interpolation process can be performed on each first-type block in the image to be processed to obtain a first interpolation result, wherein the size of the first interpolation result is larger than the size of the first-type block and equal to or greater than the size of the corresponding block in the interpolated output image. A first bidirectional predictive coding process can then be performed based on the first interpolation result to determine the motion vector corresponding to the block to be processed with the first-type block as a reference. Thus, by determining a larger interpolated output image size, the size of each block in the interpolated output image can be larger than the size of the first interpolation result corresponding to each first-type block. When storing the first interpolation result larger than the size of the first-type block, the situation of overlapping the interpolation results of other sub-blocks can be avoided to a certain extent, thereby improving the accuracy of encoding and decoding.

[0218] It should be understood that in the embodiments of the present application, the processor 1201 may be a central processing unit (CPU), and the processor 1201 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0219] In an embodiment of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the steps performed in all the above embodiments can be executed.

[0220] An embodiment of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. When the computer instructions are executed by a processor of a computer device, the methods in all the above embodiments are executed.

[0221] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The above-described program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The above-described storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0222] The above disclosure is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

[0223] It is also particularly important to note that when the above embodiments of this application are applied to specific products or technologies, if it is necessary to obtain user data, the user's permission or consent must be obtained, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A method for determining a motion vector, characterized in that: The method comprises: determining the size of an interpolation output image corresponding to the image to be processed according to the size of the image to be processed, wherein the size of the interpolation output image is larger than the size of the image to be processed; performing a first interpolation process on each first-type block in the image to be processed to obtain a first interpolation result, where the size of the first interpolation result is larger than the size of the first-type block and the size of the first interpolation result is smaller than or equal to the size of the corresponding block in the interpolation output image; A first bidirectional predictive coding process is performed according to the first interpolation result to determine a motion vector corresponding to a block to be processed with the first type of block as a reference.

2. The method according to claim 1, characterized in that The step of determining the size of the interpolated output image corresponding to the image to be processed according to the size of the image to be processed includes: Acquire the size of the first type block and the expanded size of the first type block; determining a size of an output block according to a size of the first type of block and the expanded size; The size of the interpolated output image is determined according to the number of blocks in the image to be processed and the size of the output block.

3. The method according to claim 1, characterized in that The performing a first bidirectional predictive coding process according to the first interpolation result to determine a motion vector corresponding to a block to be processed with the first type of block as a reference includes: Obtaining the motion vector of the current block to be processed at integer pixel precision; extracting a first interpolation result of a target first-type block from the interpolation output image, where the target first-type block is a reference block of the current block; performing a first sub-pixel search process based on the first interpolation result to obtain a first motion vector offset corresponding to the current block; The motion vector of the current block at integer pixel precision is corrected according to the first motion vector offset to obtain a first corrected motion vector corresponding to the current block.

4. The method according to claim 3, characterized in that The obtaining of the motion vector of the current block to be processed at integer pixel precision includes: Obtaining position information of the current block and an initial motion vector corresponding to the current block; determining, in the image to be processed, the target first-type block corresponding to the current block according to the position information of the current block and the initial motion vector; An integer pixel search process is performed on the target first-type block to obtain a motion vector of the current block at integer pixel accuracy.

5. The method according to claim 3, characterized in that The first corrected motion vector includes a forward motion vector and a backward motion vector, the interpolated output image includes the forward reference frame and the backward reference frame, and the method further includes: Determining, in the forward reference frame, a first prediction block corresponding to the current block according to the position information of the current block and the forward motion vector; Determining, in the backward reference frame, a second prediction block corresponding to the current block according to the position information of the current block and the backward motion vector; A weighted prediction process is performed on the first prediction block and the second prediction block to obtain a prediction block corresponding to the current block.

6. The method according to claim 4, characterized in that The obtaining of the initial motion vector corresponding to the current block includes: Decoding the video code stream to obtain video coding parameters, wherein the video coding parameters include a motion vector index of the current block; According to the motion vector index, a candidate motion vector indicated by the motion vector index is determined from a set candidate motion vector list to serve as an initial motion vector corresponding to the current block.

7. The method according to claim 3, characterized in that The method further comprises: performing a second interpolation process on the target first-type block to obtain a second interpolation result of the target first-type block, wherein a size of the second interpolation result of the target first-type block is the same as a size of the target first-type block; Performing a second sub-pixel search process based on a second interpolation result of the target first-type block to obtain a second motion vector offset corresponding to the current block; Correcting the motion vector of the current block at integer pixel precision according to the second motion vector offset to obtain a second corrected motion vector corresponding to the current block; An optimal motion vector corresponding to the current block is determined according to the first corrected motion vector and the second corrected motion vector.

8. The method according to claim 7, characterized in that Determining the optimal motion vector corresponding to the current block according to the first corrected motion vector and the second corrected motion vector includes: A motion vector that meets a preset error condition is selected from the first corrected motion vector and the second corrected motion vector, and the selected motion vector is used as the optimal motion vector corresponding to the current block.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: performing a second interpolation process on each second-type block in the image to be processed to obtain a second interpolation result, where the size of the second interpolation result is the same as the size of the second-type block; A second bidirectional predictive coding process is performed according to the second interpolation result to determine a motion vector corresponding to a block to be processed with the second type block as a reference.

10. The method according to claim 9, characterized in that In the process of performing the first interpolation process on each first type block in the image to be processed to obtain the first interpolation result, the second interpolation process is simultaneously performed on each second type block in the image to be processed to obtain the second interpolation result.

11. The method according to claim 9, characterized in that The first interpolation process is performed on each of the first type blocks in the image to be processed, and the second interpolation process is performed on each of the second type blocks in the image to be processed, using a graphics processor (GPU) of a video processing device.

12. A device for determining a motion vector, characterized in that: include: a determining unit, configured to determine a size of an interpolated output image corresponding to the image to be processed according to a size of the image to be processed, wherein the size of the interpolated output image is larger than the size of the image to be processed; A processing unit is configured to perform a first interpolation process on each first-type block in the image to be processed to obtain a first interpolation result, where the size of the first interpolation result is larger than the size of the first-type block and the size of the first interpolation result is smaller than or equal to the size of the corresponding block in the interpolation output image; and perform a first bidirectional predictive coding process based on the first interpolation result to determine a motion vector corresponding to the block to be processed with each first-type block as a reference.

13. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more computer programs, which, when executed by the one or more processors, enables the electronic device to implement the method for determining a motion vector according to any one of claims 1 to 11.

14. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for determining a motion vector according to any one of claims 1 to 11 is implemented.

15. A computer program product, characterized in that The computer program product includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device executes the method for determining a motion vector according to any one of claims 1 to 11.

Images