Image processing method, device, equipment and readable storage medium

Abstract

The embodiment of the present application provides an image processing method, apparatus, device and readable storage medium, wherein the method includes: determining an extended image block of a current image block from a video image frame, wherein the extended image block includes the current image block; verifying the initial motion vector of the current image block using the extended image block and the reference image frame of the video image frame to obtain a verification result; wherein the initial motion vector of the current image block is obtained by processing the reference image frame using a first image search method; and determining the target motion vector of the current image block according to the initial motion vector or the second image search method using the verification result. Through the image processing method provided by the embodiment of the present application, the target motion vector can be determined according to the verification result of the initial motion vector, thereby effectively reducing the algorithm complexity and improving the video coding efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to image processing methods, devices, equipment and readable storage media. Background Art

[0002] With the development of computer technology, more and more videos are transmitted on the network. In the process of video transmission, in order to save network resources, the video is usually encoded. In the process of video encoding, in order to reduce the amount of data, motion estimation technology is usually used; motion estimation technology refers to the technology of estimating the motion displacement between two frames of images. Motion estimation technology can effectively remove the redundancy between video frames, so that the amount of data transmitted by the video is greatly reduced.

[0003] The commonly used motion estimation method is to first search the reference image frame using an integer pixel precision search method, and then search the reference image frame using a sub-pixel precision search method to obtain the motion vector of the image block. However, this motion estimation method has high algorithm complexity and large amount of data calculation, which easily leads to low video coding efficiency. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, device and readable storage medium, which can determine the target motion vector according to the verification result of the initial motion vector, thereby effectively reducing the algorithm complexity and improving the video encoding efficiency.

[0005] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Determine an extended image block of a current image block from a video image frame, wherein the extended image block includes the current image block;

[0007] Using the extended image block and the reference image frame of the video image frame to verify the initial motion vector of the current image block, and obtain a verification result; wherein the initial motion vector of the current image block is obtained by processing the reference image frame using a first image search method;

[0008] The target motion vector of the current image block is determined using the verification result according to the initial motion vector or a second image search method.

[0009] On the one hand, an embodiment of the present application provides an image processing device, the device comprising:

[0010] An extension unit, configured to determine an extended image block of a current image block from a video image frame, wherein the extended image block includes the current image block;

[0011] a processing unit, configured to verify the initial motion vector of the current image block by using the extended image block and the reference image frame of the video image frame to obtain a verification result; wherein the initial motion vector of the current image block is obtained by processing the reference image frame by using a first image search method;

[0012] A determination unit is used to determine the target motion vector of the current image block according to the initial motion vector or a second image search method using the verification result.

[0013] On the one hand, an embodiment of the present application provides a computer device, comprising: a processor, a communication interface and a memory, wherein the processor, the communication interface and the memory are interconnected, wherein the memory stores an executable program code, and the processor is used to call the executable program code to implement the image processing method provided in the embodiment of the present application.

[0014] Accordingly, an embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer implements the image processing method provided by the embodiment of the present application.

[0015] Accordingly, the embodiment of the present application further provides a computer program product, the computer program product includes a computer program or a computer instruction, and the computer program or the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer program or the computer instruction from the computer-readable storage medium, and the processor executes the computer program or the computer instruction, so that the computer device implements the image processing method provided in the embodiment of the present application.

[0016] The image processing method provided by the present application can determine the extended image block of the current image block from the video image frame, and use the extended image block and the reference image frame of the video image frame to verify the initial motion vector of the current image block; the target motion vector of the current image block can be determined according to the initial motion vector or the second image search method based on the verification result. Through the image processing method provided by the embodiment of the present application, the extended image block containing the current image block can be determined, so as to facilitate the verification of the initial motion vector using the extended image block; the initial motion vector of the current image block can be verified using the extended image block and the reference image frame, so as to determine whether the initial motion vector meets the requirements; the target motion vector can be determined according to the initial motion vector or the second image search method based on the verification result. When the target motion vector is determined according to the initial motion vector, it is not necessary to use the second image search method for search processing, thereby effectively reducing the complexity of the algorithm and reducing the amount of calculation. At the same time, it can also effectively shorten the time to determine the target motion vector, thereby improving the efficiency of video encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 is a schematic diagram of the system architecture of an image processing system provided in an embodiment of the present application;

[0019] Figure 2 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0020] Figure 3 is a flowchart of another image processing method provided in an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of an extended image block provided in an embodiment of the present application;

[0022] Figure 5 This is an implementation method for determining a matching image block provided by an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of an image processing method provided in an embodiment of the present application;

[0024] Figure 7 is a structural block diagram of an image processing device provided in an embodiment of the present application;

[0025] Figure 8 It is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] It should be noted that the descriptions of "first", "second", etc. involved in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the technical features defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0028] When performing video encoding, motion estimation technology is usually used to remove redundancy between video frames, which greatly reduces the amount of data transmitted by the video. The commonly used motion estimation method is to first search the reference image frame using an integer pixel precision search method, and then search the reference image frame using a sub-pixel precision search method to obtain the motion vector of the current image block. However, this method of determining the motion vector has a high algorithm complexity, a large amount of data calculation, and is likely to lead to low video encoding efficiency.

[0029] Based on this, an embodiment of the present application provides an image processing method, which can determine an extended image block of a current image block from a video image frame, wherein the extended image block includes the current image block; the initial motion vector of the current image block is verified using the extended image block and the reference image frame of the video image frame to obtain a verification result; wherein the initial motion vector of the current image block is obtained by processing the reference image frame using a first image search method; and the target motion vector of the current image block is determined using the verification result according to the initial motion vector or the second image search method. Through the method provided by the embodiment of the present application, the target motion vector can be determined according to the initial motion vector or the second image search method using the verification result, which can effectively reduce the algorithm complexity, reduce the amount of data calculation, and improve the video encoding efficiency.

[0030] The image processing method provided in the embodiment of the present application can also be applied to the field of cloud computing. Cloud computing is a computing mode that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices. The image processing method provided in the embodiment of the present application can be implemented by a cloud computing device: when performing video encoding, the video to be encoded can be sent to the cloud computing device. For the video image frame in the video to be encoded, the cloud computing device can determine the extended image block of the current image block from the video image frame, and use the extended image block and the reference image frame to verify the initial motion vector of the current image block to obtain a verification result, and use the verification result to determine the target motion vector of the current image block according to the initial motion vector or the second image search method, thereby effectively reducing the complexity of the algorithm and improving the efficiency of video encoding.

[0031] The architecture of the image processing system provided in the embodiments of the present application will be introduced below in conjunction with the accompanying drawings.

[0032] See also Figure 1, which is a schematic diagram of the system architecture of an image processing system provided in an embodiment of the present application, the image processing system includes a terminal device 101, an image processing server 102 and a database 103, and the image processing server 102 can exchange data with the terminal device 101 and the database 103. Among them:

[0033] The terminal device 101 can generate and send a video acquisition request, and can also receive encoded video data, and decode the encoded video data to obtain the video to be encoded, and display the video to be encoded. The terminal device 101 can be a handheld device (such as a smart phone, a tablet computer) with video decoding function and communication function, a computing device (such as a personal computer (Personal Computer, PC), a vehicle terminal, an intelligent voice interaction device, a wearable device or other intelligent device, but is not limited thereto.

[0034] The image processing server 102 can receive a video acquisition request sent by the terminal device 101, determine the video to be encoded according to the video acquisition request, and use the image processing method provided by the present application to determine the target motion vector of the image block in each video image frame, thereby obtaining the encoded video data corresponding to the video to be encoded; the encoded video data can be sent to the terminal device 101; the image processing server 102 can also obtain the search method related data of the image search method from the database 103. The image processing server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0035] The database 103 is used to store search method related data of the first image search method and the second image search method, for example: search parameters of the first image search method, similarity measurement data, etc. The database 103 can be a local database in the image processing server 102, or a cloud database associated with the image processing server 102 (i.e., a database deployed in the cloud). Specifically, it can be deployed based on any one of private cloud, public cloud, hybrid cloud, edge cloud, etc., so that the cloud database focuses on different functions. For example, the database deployed in the private cloud has the user's personal equipment as the basic cloud hardware, and focuses more on serving a small number of users, while the database deployed in the public cloud is deployed based on the cloud platform provided by a third party, which allows the data stored in the database to achieve data sharing, and any user's data can be stored in the database, and any user can also use the data in the database.

[0036] The following will explain in detail Figure 1 The working principle of the image processing system shown:

[0037] The terminal device 101 can send a video acquisition request to the image processing server 102, and the image processing server 102 can determine the video to be encoded according to the received video acquisition request. The image processing server 102 can obtain the search method related data of the first image search method and the second image search method from the database 103. For any video image frame in the video to be encoded, the image processing server 102 can process the reference image frame of the video image frame using the first image search method according to the related data of the first image search method to obtain the initial motion vector of the current image block; the image processing server 102 can determine the extended image block of the current image block from the video image frame, and the extended image block includes the current image block; the image processing server 102 can use the extended image block and the reference image frame to verify the initial motion vector of the current image block to obtain a verification result; if the verification result indicates that the verification is passed, the target motion vector can be directly determined according to the initial motion vector, and if the verification result indicates that the verification is not passed, the target motion vector can be determined according to the second image search method, thereby realizing the video encoding processing of the video to be encoded and obtaining the encoded video data. The image processing server 102 sends the encoded video data to the terminal device 101, and the terminal device 101 can perform video decoding processing on the encoded video data to obtain the video to be encoded. Through the image processing method provided in the embodiment of the present application, the initial motion vector can be verified and the target motion vector can be determined according to the verification result, which effectively reduces the calculation complexity of the algorithm, reduces the amount of data calculation, and improves the efficiency of video encoding.

[0038] It is to be understood that the schematic diagram of the image processing system described in the embodiment of the present application is to more clearly illustrate the image processing method of the embodiment of the present application, and does not constitute a limitation on the image processing method provided by the embodiment of the present application. For example, the image processing method provided by the embodiment of the present application can be executed by the image processing server 102, and can also be executed by other devices that are different from the image processing server 102 and can communicate with the terminal device 101 and the database 103. It is known to those skilled in the art that Figure 1 The number of terminal devices 101, image processing servers 102, and databases 103 is merely illustrative. Any number of devices may be configured according to business implementation requirements. Moreover, with the evolution of system architecture and the emergence of new business scenarios, the image processing method provided in the embodiment of the present application is also applicable to similar technical problems.

[0039] It should be noted that the collection and processing of relevant data (for example, video image frames, reference image frames) in this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0040] See also Figure 2 , Figure 2 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application. The image processing method can be implemented by the above-mentioned image processing server 102, or by other devices. The flow chart of the image processing method provided in an embodiment of the present application includes but is not limited to:

[0041] S201. Determine an extended image block of a current image block from a video image frame, where the extended image block includes the current image block.

[0042] In an embodiment of the present application, a video image frame is included in a video to be encoded, and the video to be encoded includes multiple video image frames. When performing video encoding processing on the video to be encoded, in order to reduce the amount of data, a motion estimation technology can be used to calculate the motion vector corresponding to each video image frame. When calculating the motion vector corresponding to the video image frame, the current image block can be determined from the video image frame, and the current image block can include one or more pixels. After determining the current image block, an extended image block can be determined based on the current image block, and the extended image block includes the current image block, that is, the extended image block includes the current image block and a part of the pixels around the current image block. Through the method provided in the embodiment of the present application, the extended image block can be determined, so that the initial motion vector of the current image block can be verified according to the extended image block, and an accurate verification result can be obtained.

[0043] S202. Verify the initial motion vector of the current image block using the extended image block and the reference image frame of the video image frame to obtain a verification result; wherein the initial motion vector of the current image block is obtained by processing the reference image frame using a first image search method.

[0044] In an embodiment of the present application, a reference image frame of a video image frame can be determined from a video to be encoded, and the reference image frame is used to determine a motion vector of the video image frame. After determining the current image block in the video image frame, the reference image frame can be processed using a first image search method to obtain an initial motion vector of the current image block. In a video image frame, the motion trend of the current image block and the pixels around it is usually the same, and the extended image block includes the current image block and a part of the pixels around the current image block. Therefore, the initial motion vector of the current image block can be verified using the extended image block and the reference image frame to obtain a verification result, which can indicate to a certain extent whether the initial motion vector is accurate enough. The video image frame and the reference image frame are both included in the video to be encoded, and the video image frame is associated with the reference image frame, for example: the reference image frame can be the previous image frame of the video image frame in the video to be encoded. Through the method provided in the embodiment of the present application, the verification processing of the initial motion vector can be implemented to obtain a verification result, and the accuracy of the verification result is guaranteed, so as to determine the target motion vector according to the verification result.

[0045] In one embodiment, after determining the current image block from the video image frame, the image processing method provided by the present application can also: determine the image search range from the reference image frame according to the first image search method, and perform search processing on the image search range to obtain multiple image blocks to be selected, and the image block size of the image blocks to be selected matches the image block size of the current image block; determine the image difference data between the current image block and each of the image blocks to be selected, and the image difference data includes one or more of relative difference data, conversion difference data, and difference square sum data; use the image difference data of multiple image blocks to be selected to determine the reference image block from the multiple image blocks to be selected, and perform vector calculation on the reference image block and the current image block to obtain the initial motion vector of the current image block. After determining the current image block from the video image frame, the first image search method can be used to determine the initial motion vector of the current image block: the image search range can be determined from the reference image frame according to the first image search method, and the image search range is larger than the image range corresponding to the current image block; the image search access can be searched and processed to obtain multiple image blocks to be selected, and the image block size of each image block to be selected is the same as the image block size of the current image block; the pixel values ​​of each image block to be selected can be compared with the current image block to determine the image difference data between the current image block and each image block to be selected, and the image difference data can include one or more of relative difference data, conversion difference data, and difference square sum data, and the relative difference data, conversion difference data, and difference square sum data are all similarity measurement data.

[0046] The relative difference data, namely the sum of absolute differences (SAD), can be determined based on the absolute difference between the pixel values ​​of each pixel point in the selected pixel block and the pixel value of the corresponding pixel point in the current pixel block. The sum of absolute differences can reflect the residual time domain difference between the selected image block and the current image block to a certain extent, and the calculation complexity is low. The transformed difference data, namely the sum of absolute transformed differences (SATD), is the absolute sum of coefficients after the residual is subjected to Hadamard transformation. In most cases, SATD has a better evaluation effect than SAD, but the calculation complexity of SATD is greater than that of SAD. The sum of squared difference data, namely the sum of squared differences (SSD), can be determined based on the square difference between the pixel values ​​of each pixel point in the selected image block and the pixel value of the corresponding pixel point in the current pixel block. The sum of squared differences can better reflect the difference between the two image blocks.

[0047] The image difference data of multiple image blocks to be selected can be used to determine a reference image block from the multiple image blocks to be selected, and the reference image block is the image block with the smallest difference from the current image block among the multiple image blocks to be selected. Vector calculation can be performed on the reference image block and the current image block (for example: vector calculation is performed using the position data of each pixel point of the reference image block and the position data of the corresponding pixel point in the current image block) to obtain the initial motion vector of the current image block. Through the method provided in the embodiment of the present application, multiple different difference data can be used to determine the reference image block, which ensures the similarity between the reference image block and the current image block, and can improve the accuracy of the initial motion vector to a certain extent, thereby helping to reduce the computational complexity and improve the video coding efficiency.

[0048] It should be noted that the image difference data between the current image block and each of the candidate image blocks may also include other types of similarity measurement data, such as mean absolute difference (MAD), mean squared difference (MSD), etc. The similarity measurement data included in the image difference data may also be different depending on the application requirements of video coding.

[0049] S203: Determine the target motion vector of the current image block using the verification result according to the initial motion vector or a second image search method.

[0050] In an embodiment of the present application, the verification result can be used to determine the target motion vector of the current image block according to the initial motion vector or the second image search method, and the target motion vector can be used to implement video encoding processing of the video to be encoded. When the target motion vector of the current image block is determined according to the initial motion vector, there is no need to process the reference image frame using the second image search method, which can effectively reduce the amount of calculation and improve processing efficiency. When the target motion vector of the current image block is determined according to the second image search method, the accuracy of the target motion vector can be effectively guaranteed. For multiple video image frames in the video to be encoded, the method provided by the present application can be used to determine the target motion vector, thereby reducing the amount of calculation as a whole, reducing the computational complexity, and improving the efficiency of video encoding, thereby saving network resources.

[0051] In one embodiment, the method for implementing the target motion vector of the current image block by using the verification result according to the initial motion vector or the second image search method can be: if the verification result indicates that the verification is passed, the initial motion vector is determined as the target motion vector of the current image block; if the verification result indicates that the verification is not passed, the reference image frame is processed using the second image search method to obtain the target motion vector of the current image block. If the verification result indicates that the verification is passed, it means that the initial motion vector meets the requirements, and there is no need to perform image search processing, and the initial motion vector can be directly determined as the target motion vector of the current image block; if the verification result indicates that the verification is not passed, it means that the initial motion vector does not meet the requirements, and it is necessary to use the second image search method to perform more refined processing on the reference image frame to obtain the target motion vector of the current image block. The method provided by the embodiment of the present application can effectively reduce the computational complexity, reduce the amount of data calculation, and improve the processing efficiency of video encoding.

[0052] In one embodiment, the first image search method may be an integer pixel search method, and the second image search method may be a sub-pixel search method. The existing method for determining the motion vector is: for the current image block in the video image frame, the initial motion vector of the current image block is determined by the integer pixel search method, and then the target motion vector of the current image block is determined by the sub-pixel search method. However, the algorithm complexity of the sub-pixel search method is very high, the amount of data calculation is large, and in some cases, the sub-pixel search method may cause errors due to overly detailed search. In the method provided by the present application, the initial motion vector of the current image block can be determined by the integer pixel search method, and the extended image block of the current image block can be determined. The initial motion vector is verified using the extended image block and the reference image frame to obtain a verification result; if the verification result indicates that the verification is passed, the initial motion vector is directly determined as the target motion vector, which is equivalent to terminating the determination process of the target motion vector in advance, and there is no need to perform the sub-pixel search method, thereby effectively reducing the amount of data calculation, reducing the calculation complexity, and effectively avoiding errors caused by overly detailed search, ensuring the accuracy of the motion vector, and improving the effect of video encoding.

[0053] It should be noted that the image processing method provided in the embodiment of the present application can be applied to encoders based on different coding standards (for example: H.264, H.265 and other coding standards), so that when the encoder encodes the video, it can use the method provided in the embodiment of the present application to quickly and accurately determine the optimal motion vector, thereby effectively improving the efficiency of video encoding. For example: in the mode decision process of the encoder, it is necessary to determine the optimal motion vector of each image block in the image frame. After the image processing method provided in the embodiment of the present application is applied to the encoder, the encoder can first use the integer pixel search method to determine the initial motion vector of the image block, and then determine the extended image block corresponding to the image block, and use the extended image block and the reference image frame to verify the initial motion vector. If the verification is passed, the initial motion vector is determined as the optimal motion vector. Otherwise, the sub-pixel search method is used to determine the optimal motion vector of the image block; this method effectively reduces the computational complexity and reduces the amount of data calculation.

[0054] In addition to the motion estimation process applied to the mode decision of the encoder, the image processing method provided in the embodiment of the present application can also be applied to the motion estimation process in the pre-analysis, motion compensation time domain filtering and other processes. The pre-analysis technology can eliminate the redundancy of adjacent pixels in space and time in the video, and the image processing method provided in the embodiment of the present application can be applied to the motion estimation process of the pre-analysis calculation, and the verification result of the initial motion vector is determined by extending the image block, and the best motion vector and the best matching image block are determined according to the verification result, which is conducive to improving the processing efficiency of the pre-analysis.

[0055] Motion compensated time domain filtering, also known as time domain filtering technology based on motion compensation, can eliminate noise in the video, improve the subjective feeling when watching the video, and also improve the compression effect of the video. Motion compensated time domain filtering technology usually performs weighted averaging of the current video frame and the temporally adjacent video frame (i.e., the reference video frame) to smooth the noise and achieve the effect of video denoising. When performing weighted averaging, it is necessary to use motion estimation and motion compensation technology to weighted average the best matching image block with the current image block, thereby achieving a noise reduction effect in the spatial domain and improving video coding efficiency. When determining the best matching image block, the image processing method provided by the embodiment of the present application can be used, and the initial motion vector can be verified using the extended image block of the current image block to obtain a verification result, and the target motion vector is determined according to the verification result. The matching image block corresponding to the target motion vector is the best matching image block. Through the method provided by the embodiment of the present application, the amount of data calculation can be effectively reduced, the computational complexity can be reduced, and the processing efficiency of video denoising processing can be improved. For other application scenarios where the best motion vector or the best matching image block needs to be determined, the image processing method provided by the embodiment of the present application is also applicable.

[0056] Through the image processing method provided in the embodiment of the present application, the extended image block of the current image block can be reasonably determined, so as to facilitate the determination of an accurate verification result based on the extended image block; the initial motion vector can be verified using the extended image block and the reference image frame to obtain a verification result, thereby determining whether the initial motion vector meets the requirements; when the target motion vector is determined using the initial motion vector based on the verification result, the target motion vector determination process can be terminated early, thereby effectively reducing the amount of data calculation, reducing the calculation complexity, and effectively avoiding errors caused by overly detailed searches, thereby ensuring the accuracy of the motion vector, improving the effect of video encoding, and improving the processing efficiency of video encoding.

[0057] See also Figure 3 , Figure 3 A flowchart of another image processing method provided in an embodiment of the present application. The image processing method can be implemented by the above-mentioned image processing server 102, or by other devices. The process of the image processing method provided in an embodiment of the present application includes but is not limited to:

[0058] S301 : Process a reference image frame of a video image frame using a first image search method to obtain an initial motion vector of a current image block.

[0059] In the embodiment of the present application, the video image frame and the reference image frame are both included in the video to be encoded, the current image block is included in the video image frame, and the current image block may include one or more pixels. The reference image frame may be processed using the first image search method to obtain an initial motion vector of the current image block.

[0060] In one embodiment, the implementation method of determining the initial motion vector of the current image block may be: determining an image search range from a reference image frame according to a first image search method, and performing search processing on the image search range to obtain a plurality of candidate image blocks, wherein the image block size of the candidate image blocks matches the image block size of the current image block; determining image difference data between the current image block and each candidate image block, wherein the image difference data includes one or more of relative difference data, conversion difference data, and difference square sum data; determining a reference image block from the plurality of candidate image blocks using the image difference data of the plurality of candidate image blocks, and performing vector calculation on the reference image block and the current image block to obtain the initial motion vector of the current image block. After determining the current image block from the video image frame, the first image search method can be used to determine the initial motion vector of the current image block: the image search range can be determined from the reference image frame according to the first image search method, and the image search range is larger than the image range corresponding to the current image block; the image search access can be searched to obtain multiple image blocks to be selected, and the image block size of each image block to be selected is the same as the image block size of the current image block; the pixel values ​​of each image block to be selected can be compared with the current image block to determine the image difference data between the current image block and each image block to be selected, and the image difference data can include one or more of relative difference data, conversion difference data, and difference square sum data.

[0061] The image difference data of multiple image blocks to be selected can be used to determine a reference image block from the multiple image blocks to be selected, and the reference image block is the image block with the smallest difference from the current image block among the multiple image blocks to be selected. Vector calculation can be performed on the reference image block and the current image block (for example: vector calculation is performed using the position data of each pixel point of the reference image block and the position data of the corresponding pixel point in the current image block) to obtain the initial motion vector of the current image block. Through the method provided in the embodiment of the present application, multiple different difference data can be used to determine the reference image block, which ensures the similarity between the reference image block and the current image block, and can improve the accuracy of the initial motion vector to a certain extent, thereby helping to reduce the computational complexity and improve the video coding efficiency.

[0062] S302: Determine an extended image block of the current image block from the video image frame, where the extended image block includes the current image block.

[0063] In the embodiment of the present application, an extended image block can be determined based on the current image block, and the extended image block includes the current image block, that is, the extended image block includes the current image block and a portion of pixels around the current image block. Through the method provided in the embodiment of the present application, the extended image block can be determined, so that the initial motion vector of the current image block can be verified according to the extended image block to obtain an accurate verification result.

[0064] In one embodiment, the implementation method of determining the extended image block of the current image block from the video image frame can be: determining the extended parameters of the current image block; performing an extended processing on the initial image range data of the current image block in the encoded image area of ​​the video image frame according to the extended parameters to obtain the extended image range data; and determining the extended image block from the video image frame using the extended image range data. The extended parameters of the current image block (which can be represented by K) can be determined, and the extended parameters can indicate the number of extended pixels. The motion vector of the pixel point in the encoded image area of ​​the video image frame is known, so it can be extended to the encoded image area, that is, the initial image range data of the current image block can be extended in the encoded image area of ​​the video image frame according to the extended parameters to obtain the extended image range data, and the extended image block can be determined from the video image frame using the extended image range data. Through the method provided in the embodiment of the present application, the extended image block can be reasonably determined, which is convenient for the subsequent verification result of determining the initial motion vector using the extended image block.

[0065] See also Figure 4 , which is a schematic diagram of an extended image block provided in an embodiment of the present application. Figure 4 The current image block in the video image frame includes 4 pixels ( Figure 4 The image block surrounded by solid lines in the video image frame is the current image block), and the area corresponding to the solid points in the video image frame is the encoded image area of ​​the video image frame, and the area corresponding to the hollow points is the unencoded image area of ​​the video image frame. If the expansion parameter K of the current image block is determined to be 2, the initial image range data of the current image block can be expanded in the encoded area of ​​the video image frame according to the expansion parameter to obtain the expanded image range data, and the expanded image block can be determined from the video image frame using the expanded image range data. Figure 4 As shown, based on the current image block, two pixels are extended to the left and the top of the current image block to obtain an extended image block, that is, Figure 4 The image block surrounded by the dotted line includes 16 pixels, and the extended image block includes the current image block. The method provided in the embodiment of the present application can accurately determine the extended image block, which is convenient for the subsequent use of the extended image block to determine the verification result of the initial motion vector.

[0066] It should be noted that if the current image block is the first image block in the video image frame (i.e., at this time, the video image frame does not contain the encoded image area), the initial image range data of the current image block can be extended according to the extension parameter to obtain the extended image range data; the video image frame is supplemented to obtain a supplemented image frame, the supplemented image frame includes the video image frame and multiple supplemented pixel points, the pixel values ​​of the supplemented pixel points can be fixed values, and can also be determined according to the reference image frame and the video image frame. The extended image range data can be used to determine the extended image block from the supplemented image frame. Figure 4 The size and shape of the current image block and the extended image block shown are only a feasible example. In actual application, the size and shape of the current image block and the extended image block can be adjusted according to different application requirements.

[0067] In one embodiment, the extension parameter K can be determined according to the image block size of the current image block: the extension parameter K is used to indicate the number of pixels extended to the encoded image area. If the extension parameter is too large, the extended image block is too large, and the accuracy of the verification result determined according to the extended image block and the reference image frame is low; if the extension parameter is too small, the extended image block is small, and the initial motion vector cannot be accurately evaluated, which is easy to cause errors. The extension parameter K can be determined according to the image block size of the current image block. For example, if the image block size of the current image block is 4*4 (that is, the current image block includes 16 pixels), the value range of K can be 3. Through the method provided in the embodiment of the present application, the extension parameter can be reasonably determined, so that the extended image block can be effectively determined, which is conducive to accurately determining the verification result.

[0068] S303: Use the extended image block and the reference image frame to verify the initial motion vector of the current image block to obtain a verification result.

[0069] In the embodiment of the present application, the initial motion vector can be verified, which is conducive to the early termination of the processing flow: the method of determining the target motion vector is to first use the first image search method for processing, and then use the second image search method for processing, and the method provided by the present application can use the extended image block and the reference image frame to verify the initial motion vector of the current image block to obtain a verification result, which can indicate to a certain extent whether the initial motion vector is accurate enough. It can be decided whether to use the second image search method for processing based on the verification result, that is, the early termination of the target motion vector is achieved by using the judgment process based on the verification result. Through the method provided in the embodiment of the present application, the verification process of the initial motion vector can be realized, and the verification result can be obtained, which ensures the accuracy of the verification result, so as to determine the target motion vector according to the verification result.

[0070] In one embodiment, the verification process of the initial motion vector of the current image block using the extended image block and the reference image frame of the video image frame to obtain the verification result can be implemented as follows: determining a matching image block from the reference image frame according to the extended image block and the initial motion vector; performing similarity calculation using the extended image block and the matching image block to obtain the similarity between the extended image block and the matching image block; if the similarity is greater than a similarity threshold, determining the verification result of the initial motion vector as passed verification; if the similarity is less than or equal to the similarity threshold, determining the verification result of the initial motion vector as failed verification. When performing the verification process, a matching image block can be determined from the reference image frame according to the extended image block and the initial motion vector, the matching image block is included in the reference image frame, and the image block size of the extended image block is the same as the image block size of the matching image block. The extended image block includes the current image block and a part of the pixels around the current image block. The motion trends of the current image block and the pixels around it are roughly the same, that is, the motion vectors are roughly the same. The initial motion vector of the current image block can be determined as the motion vector of the extended image block. According to the motion vector of the extended image block (that is, the initial motion vector), the matching image block can be determined from the reference image frame. If the similarity between the matching image block and the extended image block is high, it means that the initial motion vector can better represent the motion trend of the extended image block and the initial motion vector is relatively accurate. If the similarity between the matching image block and the extended image block is low, it means that the initial motion vector cannot well represent the motion trend of the extended image block and the initial motion vector is inaccurate.

[0071] Therefore, after determining the matching image block, the extended image block and the matching image block can be used to perform similarity calculation to obtain the similarity between the extended image block and the matching image block. If the similarity is greater than the similarity threshold, it means that the initial motion vector is relatively accurate, and the verification result of the initial motion vector can be determined as verification passed; if the similarity is less than or equal to the similarity threshold, it means that the initial motion vector is inaccurate, and the verification result of the initial motion vector can be determined as verification failed. Through the method provided in the embodiment of the present application, the verification process of the initial motion vector can be implemented, and the verification result of the initial motion vector can be accurately determined, which is conducive to reducing the amount of data calculation, reducing the computational complexity, and improving the efficiency of video coding.

[0072] In one embodiment, the implementation method of determining the matching image block from the reference image frame according to the extended image block and the initial motion vector can be: for any pixel point among the multiple pixels included in the extended image block, vector calculation is performed on the original position data of any pixel point according to the initial motion vector to obtain the matching position data of any pixel point; the matching pixel point of any pixel point is determined from the reference image frame according to the matching position data of any pixel point; after the matching pixel points of the multiple pixels included in the extended image block are determined, the image block composed of the matching pixel points of the multiple pixels is determined as the matching image block. The extended image block may include multiple pixels. For any pixel point among the multiple pixels included in the extended image block, vector calculation can be performed on the original position data of any pixel point according to the initial motion vector to obtain the matching position data of any pixel point. For example, if the original position data of a pixel point in the extended image block is (0.1, 0.3), the direction of the initial motion vector is from the pixel point in the video image frame to the pixel point in the reference video frame, and the magnitude of the initial motion vector is (0.1, 0.1), the matching position data of the pixel point can be determined to be (0.2, 0.4) according to the initial motion vector.

[0073] After determining the matching position data of any pixel point, the matching pixel point of any pixel point can be determined from the reference image frame according to the matching position data, and the position data of the matching pixel point in the reference image frame is the matching position data. For example, if the matching position data of a certain pixel point is (0.2, 0.4), the matching pixel point of the pixel point can be determined from the reference image frame, and the position data of the matching pixel point in the reference image frame is (0.2, 0.4). After determining the matching pixel points of the multiple pixels included in the extended image block, the pixel block composed of the matching pixel points of the multiple pixels can be determined as the matching image block. Through the method provided in the embodiment of the present application, the matching image block of the extended image block can be determined from the reference image frame using the initial motion vector, so as to facilitate the subsequent verification result of determining the initial motion vector based on the matching image block and the extended image block.

[0074] See also Figure 5 , this figure is an implementation method for determining matching image blocks provided in an embodiment of the present application. Figure 5 The video image frame and the reference image frame are included, wherein the extended image block included in the reference image frame is as described above. Figure 4As shown; the initial motion vector of the current image block in the video image frame can be determined, and the initial motion vector is recorded as MVa. The direction of the initial motion vector is from the video image frame to the reference image frame, and the size of the initial motion vector is (1,1). For pixel point A among the multiple pixel points included in the extended image block, the original position data of the pixel point is (2,3). The original position data of pixel point A can be vector-calculated according to the initial motion vector to obtain the matching position data of pixel point A (3,4). According to the matching position data of pixel point A, the matching pixel point of pixel point A can be determined from the reference image frame, that is, pixel point B in the reference image frame (such as Figure 5 As shown, the motion vector between pixel point A and pixel point B is MVa); the above method can be used to determine the matching pixel points of each pixel point included in the extended image block, and the image block composed of the matching pixel points of each pixel point can be determined as the matching image block ( Figure 5 The image block surrounded by the dotted line in the reference image frame is the matching image block). Through the method provided in the embodiment of the present application, the matching image block of the extended image block can be accurately determined, which is convenient for the subsequent verification result of the initial motion vector determined according to the matching image block and the extended image block, thereby reducing the computational complexity and improving the processing efficiency of video encoding.

[0075] In one embodiment, the implementation method of calculating the similarity between the extended image block and the matching image block by using the extended image block and the matching image block may be: calculating the pixel values ​​of the multiple pixels included in the extended image block to obtain the pixel mean and pixel standard deviation of the extended image block; calculating the pixel values ​​of the multiple pixels included in the matching image block to obtain the pixel mean and pixel standard deviation of the matching image block; determining the brightness similarity, contrast similarity and image structure similarity according to the pixel mean and pixel standard deviation of the extended image block and the pixel mean and pixel standard deviation of the matching image block; integrating the brightness similarity, contrast similarity and image structure similarity to obtain the similarity between the extended image block and the matching image block. When determining the similarity between the extended image block and the matching image block, the pixel values ​​of the multiple pixels included in the extended image block may be calculated to obtain the pixel mean and pixel standard deviation of the extended image block. The extended image block may be represented by x, and the pixel mean of the extended image block may be represented by μ x , the pixel standard deviation of the extended image block can be expressed as δ x ; The pixel values ​​of multiple pixels included in the matching image block can be calculated to obtain the pixel mean and pixel standard deviation of the matching image block. The matching image block can be represented by y, and the pixel mean of the matching image block can be represented by μ y , the pixel standard deviation of the matching image patch can be expressed as δ y .

[0076] According to the pixel mean and pixel standard deviation value of the extended image block, and the pixel mean and pixel standard deviation value of the matching image block, the brightness similarity, contrast similarity and image structure similarity can be determined, wherein the brightness similarity can reflect the similarity between the extended image block and the matching image block in the image brightness dimension, and the brightness similarity can be expressed as l(x, y); the contrast similarity can reflect the similarity between the extended image block and the matching image block in the image contrast dimension, and the contrast similarity can be expressed as c(x, y); the image structure similarity can reflect the similarity between the extended image block and the matching image block in the image structure dimension, and the image structure similarity can be expressed as s(x, y); the brightness similarity, contrast similarity and image structure similarity can be integrated to obtain the similarity between the extended image block and the matching image block. Through the method provided in the embodiment of the present application, the similarity between the extended image block and the matching image block can be accurately determined, so that the verification result of the initial motion vector can be determined according to the similarity, which is conducive to improving the accuracy of the image processing method, reducing the computational complexity, and improving the video encoding efficiency.

[0077] In one embodiment, a structural similarity index (Structure Similarity Index Measure, SSIM) can be used as a similarity measurement indicator between the extended image block and the matching image block: the pixel mean of the extended image block can be calculated as μ x , the pixel standard deviation of the expanded image block is δ x ; The pixel mean of the matching image block is μ y , the pixel standard deviation of the matching image patch is δ y , the pixel covariance between the expanded image patch and the matched image patch is δ xy , the method for determining brightness similarity, contrast similarity and image structure similarity can be shown as the following equations (1), (2) and (3):

[0078]

[0079]

[0080] s(x,y)=(δ xy +C3) / (δ x *δ y +C3) (3)

[0081] Among them, l(x, y) is brightness similarity, c(x, y) is contrast similarity, s(x, y) is image structure similarity, and C1, C2, and C3 are all constants. Through the above three formulas, the brightness similarity, contrast similarity, and image structure similarity between the extended image block and the matching image block can be determined. According to the brightness similarity, contrast similarity, and image structure similarity, the implementation method of determining the similarity between the extended image block and the matching image block can be shown in the following formula (4):

[0082] SSIM(x,y)=[l(x,y)] α *[c(x,y)] β *[s(x,y)] γ (4)

[0083] Among them, α, β, and γ are all constant parameters greater than 0. Through the above formula, the SSIM similarity between the extended image block and the matching image block can be determined. The SSIM similarity takes into account the structural information of the two image blocks, can more accurately reflect the subjective similarity of the two image blocks, and has high calculation efficiency. Through the method provided in the embodiment of the present application, the similarity between the extended image block and the matching image block can be accurately determined, so that the verification result of the initial motion vector can be determined based on the similarity.

[0084] S304: Determine the target motion vector of the current image block using the verification result according to the initial motion vector or a second image search method.

[0085] In an embodiment of the present application, the verification result can be used to determine the target motion vector of the current image block according to the initial motion vector or the second image search method, and the target motion vector can be used to implement video encoding processing of the video to be encoded. When the target motion vector of the current image block is determined according to the initial motion vector, there is no need to process the reference image frame using the second image search method, which can effectively reduce the amount of calculation and improve processing efficiency. When the target motion vector of the current image block is determined according to the second image search method, the accuracy of the target motion vector can be effectively guaranteed. For multiple video image frames in the video to be encoded, the method provided by the present application can be used to determine the target motion vector, thereby reducing the amount of calculation as a whole, reducing the computational complexity, and improving the efficiency of video encoding, thereby saving network resources.

[0086] In one embodiment, the method for implementing the target motion vector of the current image block by using the verification result according to the initial motion vector or the second image search method can be: if the verification result indicates that the verification is passed, the initial motion vector is determined as the target motion vector of the current image block; if the verification result indicates that the verification is not passed, the reference image frame is processed using the second image search method to obtain the target motion vector of the current image block. If the verification result indicates that the verification is passed, it means that the initial motion vector meets the requirements, and there is no need to perform image search processing, and the initial motion vector can be directly determined as the target motion vector of the current image block; if the verification result indicates that the verification is not passed, it means that the initial motion vector does not meet the requirements, and it is necessary to use the second image search method to perform more refined processing on the reference image frame to obtain the target motion vector of the current image block. The method provided by the embodiment of the present application can effectively reduce the computational complexity, reduce the amount of data calculation, and improve the processing efficiency of video encoding.

[0087] In one embodiment, the first image search method may be an integer pixel search method, and the second image search method may be a sub-pixel search method. The existing method for determining the motion vector is: for the current image block in the video image frame, the integer pixel search method is used to determine the initial motion vector of the current image block, and then the sub-pixel search method is used to determine the target motion vector of the current image block. However, the sub-pixel search method has a high algorithm complexity and a large amount of data calculation. In some cases, the sub-pixel search method may cause errors due to overly detailed search. In the method provided by the present application, the integer pixel search method can be used to determine the initial motion vector of the current image block, and the extended image block of the current image block can be determined. The initial motion vector is verified using the reference image frame of the extended image block to obtain a verification result; if the verification result indicates that the verification is passed, the initial motion vector is directly determined as the target motion vector, which is equivalent to terminating the target motion vector determination process in advance, and there is no need to perform the sub-pixel search method, thereby effectively reducing the amount of data calculation, reducing the calculation complexity, and effectively avoiding errors caused by overly detailed search, ensuring the accuracy of the motion vector, and improving the effect of video encoding.

[0088] In one embodiment, if the verification result indicates that the verification fails, the reference image frame is processed using the second image search method to obtain the target motion vector of the current image block, which can be implemented as follows: interpolation processing is performed on the sub-pixel position that may be searched in the reference image frame to obtain an interpolated image frame, that is, the reference image frame is up-sampled using an interpolation method (such as bilinear interpolation or a sub-pixel motion compensation method specified in the coding standard) to obtain an image frame with a higher resolution, which allows subsequent searches to be performed at sub-pixel positions; after determining the interpolated image frame, the interpolated image frame can be searched at a sub-pixel level in a small area around the whole pixel search result (for example: traversing possible positions in a certain order, and calculating distortion values ​​and rate-distortion cost values ​​respectively); according to the sub-pixel level search results, the target image block with the lowest rate-distortion cost (that is, the highest similarity with the current image block) is determined from the sub-pixel search area, and vector calculation is performed based on the target image block and the current image block to obtain the target motion vector of the current image block. Through the method provided in the embodiment of the present application, the target motion vector can be accurately determined, which is conducive to improving the accuracy of the target motion vector.

[0089] See also Figure 6 , which is a schematic diagram of an image processing method provided by an embodiment of the present application. When performing video encoding, the motion vector of the video image frame can be calculated. For the current image block of the video image frame, the reference image frame can be processed by the first image search method to obtain the initial motion vector of the current image block; the extended image block of the current image block can be determined from the video image frame, and the matching image block can be determined from the reference image frame using the extended image block and the initial motion vector, and the similarity between the matching image block and the extended image block is calculated; it is determined whether the similarity is greater than the similarity threshold, if so, it is determined that the initial motion vector verification is passed, and the initial motion vector can be determined as the target motion vector of the current image block, if not, it is determined that the initial motion vector verification is not passed, and the reference image frame can be processed by the second image search method to obtain the target motion vector of the current image block. Through the method provided by the embodiment of the present application, the early termination of the determination process of the target motion vector can be achieved, thereby effectively reducing the amount of data calculation, reducing the calculation complexity, and effectively avoiding the errors caused by overly detailed search, ensuring the accuracy of the target motion vector, and improving the effect of video encoding.

[0090] Through the image processing method provided by the embodiment of the present application, a variety of different types of similarity measurement data can be used to determine the initial motion vector of the current image block, thereby improving the accuracy of the initial motion vector and reducing the amount of data calculation; the extended image block of the current image block can be reasonably determined, so as to facilitate determining an accurate verification result based on the extended image block; the initial motion vector can be verified using the extended image block and the reference image frame to obtain a verification result, thereby determining whether the initial motion vector meets the requirements; when the target motion vector is determined using the initial motion vector based on the verification result, the target motion vector determination process can be terminated early, thereby effectively reducing the amount of data calculation, reducing the calculation complexity, and effectively avoiding errors caused by overly detailed searches, thereby ensuring the accuracy of the motion vector, improving the effect of video encoding, and improving the processing efficiency of video encoding.

[0091] See also Figure 7 , Figure 7 This is a structural block diagram of an image processing device provided in an embodiment of the present application. The device includes:

[0092] An extension unit 701 is configured to determine an extended image block of a current image block from a video image frame, wherein the extended image block includes the current image block;

[0093] The processing unit 702 is configured to perform verification processing on the initial motion vector of the current image block using the extended image block and the reference image frame of the video image frame to obtain a verification result; wherein the initial motion vector of the current image block is obtained by processing the reference image frame using a first image search method;

[0094] The determination unit 703 is configured to determine the target motion vector of the current image block by using the verification result according to the initial motion vector or a second image search method.

[0095] In one embodiment, when the determination unit 703 uses the verification result to determine the target motion vector of the current image block according to the initial motion vector or the second image search method, it is specifically used to: if the verification result indicates that the verification is passed, determine the initial motion vector as the target motion vector of the current image block; if the verification result indicates that the verification is not passed, use the second image search method to process the reference image frame to obtain the target motion vector of the current image block.

[0096] In one embodiment, when determining an extended image block of a current image block from a video image frame, the extension unit 701 is specifically used to: determine an extension parameter of the current image block; perform an extension process on the initial image range data of the current image block in an encoded image area of ​​the video image frame according to the extension parameter to obtain extended image range data; and determine the extended image block from the video image frame using the extended image range data.

[0097] In one embodiment, when the processing unit 702 uses the extended image block and the reference image frame of the video image frame to verify the initial motion vector of the current image block and obtain the verification result, it is specifically used to: determine a matching image block from the reference image frame according to the extended image block and the initial motion vector; perform similarity calculation using the extended image block and the matching image block to obtain the similarity between the extended image block and the matching image block; if the similarity is greater than a similarity threshold, determine that the verification result of the initial motion vector is verification passed; if the similarity is less than or equal to the similarity threshold, determine that the verification result of the initial motion vector is verification failed.

[0098] In one embodiment, when determining a matching image block from the reference image frame according to the extended image block and the initial motion vector, the processing unit 702 is specifically configured to: for any pixel point among a plurality of pixel points included in the extended image block, perform vector calculation on the original position data of the any pixel point according to the initial motion vector to obtain the matching position data of the any pixel point; determine the matching pixel point of the any pixel point from the reference image frame according to the matching position data of the any pixel point; and after determining the matching pixel points of the plurality of pixel points included in the extended image block, determine the image block composed of the matching pixel points of the plurality of pixel points as the matching image block.

[0099] In one embodiment, when the processing unit 702 uses the extended image block and the matching image block to perform similarity calculation to obtain the similarity between the extended image block and the matching image block, it is specifically used to: calculate the pixel values ​​of multiple pixels included in the extended image block to obtain the pixel mean and pixel standard deviation value of the extended image block; calculate the pixel values ​​of multiple pixels included in the matching image block to obtain the pixel mean and pixel standard deviation value of the matching image block; determine the brightness similarity, contrast similarity and image structure similarity according to the pixel mean and pixel standard deviation value of the extended image block and the pixel mean and pixel standard deviation value of the matching image block; integrate the brightness similarity, the contrast similarity and the image structure similarity to obtain the similarity between the extended image block and the matching image block.

[0100] In one embodiment, the expansion unit 701 is further used to: determine an image search range from the reference image frame according to the first image search method, and perform search processing on the image search range to obtain multiple candidate image blocks, wherein the image block size of the candidate image blocks matches the image block size of the current image block; determine image difference data between the current image block and each of the candidate image blocks, wherein the image difference data includes one or more of relative difference data, conversion difference data, and difference square sum data; determine a reference image block from the multiple candidate image blocks using the image difference data of the multiple candidate image blocks, and perform vector calculation on the reference image block and the current image block to obtain an initial motion vector of the current image block.

[0101] It is understandable that the functions of each functional unit of the image processing device of the embodiment of the present application can be specifically implemented according to the image processing method in the above-mentioned method embodiment, and its specific implementation process can refer to the relevant description in the above-mentioned image processing method embodiment, which will not be repeated here. In the embodiment of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that contains the function of the module or unit.

[0102] Through the image processing device provided by the embodiment of the present application, a variety of different types of similarity measurement data can be used to determine the initial motion vector of the current image block, thereby improving the accuracy of the initial motion vector and reducing the amount of data calculation; the extended image block of the current image block can be reasonably determined, so as to facilitate determining an accurate verification result based on the extended image block; the initial motion vector can be verified using the extended image block and the reference image frame to obtain a verification result, thereby determining whether the initial motion vector meets the requirements; when the target motion vector is determined using the initial motion vector based on the verification result, the target motion vector determination process can be terminated early, thereby effectively reducing the amount of data calculation, reducing the calculation complexity, and effectively avoiding errors caused by overly detailed searches, thereby ensuring the accuracy of the motion vector, improving the effect of video encoding, and improving the processing efficiency of video encoding.

[0103] See also Figure 8 , Figure 8A block diagram of a computer device provided in an embodiment of the present application. The computer device described in the embodiment of the present application includes: a processor 801, a communication interface 802, and a memory 803. The processor 801, the communication interface 802, and the memory 803 may be connected via a bus or other means, and the embodiment of the present application takes the connection via a bus as an example.

[0104] Among them, the processor 801 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which can parse various instructions in the computer device and process various data of the computer device. For example, the CPU can be used to parse the power on and off instructions sent by the user to the computer device, and control the computer device to perform power on and off operations; for another example, the CPU can transmit various interactive data between the internal structures of the computer device, and so on. The communication interface 802 can optionally include a standard wired interface, a wireless interface (such as Wi-Fi, a mobile communication interface, etc.), which is controlled by the processor 801 to send and receive data. The memory 803 (Memory) is a memory device in the computer device for storing programs and data. It can be understood that the memory 803 here can include both the built-in memory of the computer device and the extended memory supported by the computer device. The memory 803 provides a storage space, which stores the operating system of the computer device, which may include but is not limited to: Android system, iOS system, Windows Phone system, etc., and this application does not limit this.

[0105] In the embodiment of the present application, the processor 801 performs the following operations by running the executable program code in the memory 803:

[0106] Determine an extended image block of a current image block from a video image frame, wherein the extended image block includes the current image block;

[0107] Using the extended image block and the reference image frame of the video image frame to verify the initial motion vector of the current image block, and obtain a verification result; wherein the initial motion vector of the current image block is obtained by processing the reference image frame using a first image search method;

[0108] The target motion vector of the current image block is determined using the verification result according to the initial motion vector or a second image search method.

[0109] In one embodiment, when the processor 801 uses the verification result to determine the target motion vector of the current image block according to the initial motion vector or the second image search method, it is specifically used to: if the verification result indicates that the verification is passed, determine the initial motion vector as the target motion vector of the current image block; if the verification result indicates that the verification is not passed, use the second image search method to process the reference image frame to obtain the target motion vector of the current image block.

[0110] In one embodiment, when determining an extended image block of a current image block from a video image frame, the processor 801 is specifically used to: determine an extended parameter of the current image block; perform an extended processing on the initial image range data of the current image block in an encoded image area of ​​the video image frame according to the extended parameter to obtain extended image range data; and determine the extended image block from the video image frame using the extended image range data.

[0111] In one embodiment, when the processor 801 uses the extended image block and the reference image frame of the video image frame to verify the initial motion vector of the current image block and obtain a verification result, it is specifically used to: determine a matching image block from the reference image frame based on the extended image block and the initial motion vector; perform similarity calculation using the extended image block and the matching image block to obtain the similarity between the extended image block and the matching image block; if the similarity is greater than a similarity threshold, determine that the verification result of the initial motion vector is verification passed; if the similarity is less than or equal to the similarity threshold, determine that the verification result of the initial motion vector is verification failed.

[0112] In one embodiment, when the processor 801 determines the matching image block from the reference image frame based on the extended image block and the initial motion vector, it is specifically used to: for any pixel point among the multiple pixel points included in the extended image block, perform vector calculation on the original position data of the any pixel point according to the initial motion vector to obtain the matching position data of the any pixel point; determine the matching pixel point of the any pixel point from the reference image frame according to the matching position data of the any pixel point; after determining the matching pixel points of the multiple pixel points included in the extended image block, determine the image block composed of the matching pixel points of the multiple pixel points as the matching image block.

[0113] In one embodiment, when the processor 801 uses the extended image block and the matching image block to perform similarity calculation to obtain the similarity between the extended image block and the matching image block, it is specifically used to: calculate the pixel values ​​of multiple pixel points included in the extended image block to obtain the pixel mean and pixel standard deviation value of the extended image block; calculate the pixel values ​​of multiple pixel points included in the matching image block to obtain the pixel mean and pixel standard deviation value of the matching image block; determine the brightness similarity, contrast similarity and image structure similarity according to the pixel mean and pixel standard deviation value of the extended image block and the pixel mean and pixel standard deviation value of the matching image block; integrate the brightness similarity, the contrast similarity and the image structure similarity to obtain the similarity between the extended image block and the matching image block.

[0114] In one embodiment, the processor 801 is further used to: determine an image search range from the reference image frame according to the first image search method, and perform search processing on the image search range to obtain multiple candidate image blocks, wherein the image block size of the candidate image blocks matches the image block size of the current image block; determine image difference data between the current image block and each of the candidate image blocks, wherein the image difference data includes one or more of relative difference data, conversion difference data, and difference square sum data; determine a reference image block from the multiple candidate image blocks using the image difference data of the multiple candidate image blocks, and perform vector calculation on the reference image block and the current image block to obtain an initial motion vector of the current image block.

[0115] In a specific implementation, the processor 801, communication interface 802 and memory 803 described in the embodiment of the present application can execute the implementation method of the image processing server described in an image processing method provided in an embodiment of the present application, and can also execute the implementation method described in an image processing device provided in an embodiment of the present application, which will not be repeated here.

[0116] Through the computer device provided by the embodiment of the present application, a variety of different types of similarity measurement data can be used to determine the initial motion vector of the current image block, thereby improving the accuracy of the initial motion vector and reducing the amount of data calculation; the extended image block of the current image block can be reasonably determined, so as to facilitate determining an accurate verification result based on the extended image block; the initial motion vector can be verified using the extended image block and the reference image frame to obtain a verification result, thereby determining whether the initial motion vector meets the requirements; when the target motion vector is determined using the initial motion vector based on the verification result, the target motion vector determination process can be terminated early, thereby effectively reducing the amount of data calculation, reducing the calculation complexity, and effectively avoiding errors caused by overly detailed searches, thereby ensuring the accuracy of the motion vector, improving the effect of video encoding, and improving the processing efficiency of video encoding.

[0117] The present application also provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the image processing method as described in the present application. The specific implementation method can be referred to the above description, and will not be repeated here.

[0118] The embodiment of the present application also provides a computer program product, which includes a computer program or a computer instruction, and the computer program or the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer program or the computer instruction from the computer-readable storage medium, and the processor executes the computer program or the computer instruction, so that the computer device performs the image processing method as described in the embodiment of the present application. The specific implementation method can be referred to the above description, which will not be repeated here.

[0119] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0120] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a computer, a server or a network device, etc., specifically a processor in a computer device) to perform all or part of the steps of the above methods of each embodiment of the present application. Among them, the aforementioned storage medium may include: U disk, mobile hard disk, magnetic disk, optical disk, read-only memory (English: Read-Only Memory, abbreviated: ROM) or random access memory (English: Random Access Memory, abbreviated: RAM) and other media that can store program codes.

[0121] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, it should be understood that the technical solutions recorded in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Determine an extended image block of a current image block from a video image frame, wherein the extended image block includes the current image block; For any pixel point among the multiple pixel points included in the extended image block, vector calculation is performed on the original position data of the any pixel point according to the initial motion vector of the current image block to obtain the matching position data of the any pixel point; wherein the initial motion vector of the current image block is obtained by processing the reference image frame of the video image frame using the first image search method; Determining a matching pixel point of any pixel point from the reference image frame according to the matching position data of any pixel point; After determining the matching pixel points of the plurality of pixel points included in the extended image block, determining the image block composed of the matching pixel points of the plurality of pixel points as the matching image block; Calculating similarity using the extended image block and the matching image block to obtain similarity between the extended image block and the matching image block; If the similarity is greater than the similarity threshold, determining that the verification result of the initial motion vector is verification passed; If the similarity is less than or equal to the similarity threshold, determining that the verification result of the initial motion vector is verification failure; The target motion vector of the current image block is determined using the verification result according to the initial motion vector or a second image search method.

2. The method according to claim 1, characterized in that The step of using the verification result to determine the target motion vector of the current image block according to the initial motion vector or the second image search method includes: If the verification result indicates that the verification is passed, determining the initial motion vector as the target motion vector of the current image block; If the verification result indicates that the verification fails, the reference image frame is processed using a second image search method to obtain a target motion vector of the current image block.

3. The method according to claim 1, characterized in that The step of determining an extended image block of a current image block from a video image frame comprises: Determining an expansion parameter of the current image block; Performing an expansion process on the initial image range data of the current image block in the encoded image region of the video image frame according to the expansion parameter to obtain expanded image range data; An extended image block is determined from the video image frame using the extended image range data.

4. The method according to claim 1, characterized in that: The using the extended image block and the matching image block to perform similarity calculation to obtain the similarity between the extended image block and the matching image block includes: Calculating pixel values ​​of a plurality of pixel points included in the extended image block to obtain a pixel mean value and a pixel standard deviation value of the extended image block; Calculating pixel values ​​of a plurality of pixel points included in the matching image block to obtain a pixel mean value and a pixel standard deviation value of the matching image block; Determining brightness similarity, contrast similarity and image structure similarity according to the pixel mean and pixel standard deviation value of the extended image block and the pixel mean and pixel standard deviation value of the matching image block; The brightness similarity, the contrast similarity and the image structure similarity are integrated to obtain the similarity between the extended image block and the matching image block.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Determining an image search range from the reference image frame according to the first image search method, and performing search processing on the image search range to obtain a plurality of image blocks to be selected, wherein the image block sizes of the image blocks to be selected match the image block size of the current image block; Determine image difference data between the current image block and each of the to-be-selected image blocks, wherein the image difference data includes one or more of relative difference data, conversion difference data, and difference square sum data; A reference image block is determined from the multiple image blocks to be selected by using the image difference data of the multiple image blocks to be selected, and vector calculation is performed on the reference image block and the current image block to obtain an initial motion vector of the current image block.

6. An image processing device, characterized in that: The device comprises: An extension unit, configured to determine an extended image block of a current image block from a video image frame, wherein the extended image block includes the current image block; a processing unit, configured to perform vector calculation on original position data of any pixel point among a plurality of pixel points included in the extended image block according to an initial motion vector of the current image block, so as to obtain matching position data of the any pixel point; wherein the initial motion vector of the current image block is obtained by processing a reference image frame of the video image frame using a first image search method; a determining unit, configured to determine a matching pixel point of any pixel point from the reference image frame according to the matching position data of any pixel point; The determining unit is further configured to, after determining matching pixels of the plurality of pixels included in the extended image block, determine an image block composed of the matching pixels of the plurality of pixels as a matching image block; The processing unit is further configured to perform similarity calculation using the extended image block and the matching image block to obtain the similarity between the extended image block and the matching image block; The determining unit is further configured to determine that the verification result of the initial motion vector is a verification pass if the similarity is greater than a similarity threshold; The determining unit is further configured to determine that the verification result of the initial motion vector is verification failure if the similarity is less than or equal to a similarity threshold; The determination unit is further configured to determine the target motion vector of the current image block using the verification result according to the initial motion vector or a second image search method.

7. A computer device, characterized in that: include: A processor, a communication interface and a memory, wherein the processor, the communication interface and the memory are connected to each other, wherein the memory stores an executable program code, and the processor is used to call the executable program code to implement the image processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when executed on a computer, enable the computer to implement the image processing method according to any one of claims 1 to 5.

Images