Image processing method and device, equipment, medium and program product

Abstract

The embodiment of the invention provides an image processing method and device, equipment, a medium and a program product, and can relate to the technical field of encoding and decoding, the method is applied to a heterogeneous encoder or a heterogeneous decoder, and the heterogeneous encoder or the heterogeneous decoder comprises a first processor and a second processor; the method comprises the following steps that: a first processor acquires an image block to be filtered, and caches the image block to be filtered to a first cache shared by the first processor and a second processor; the second processor reads the to-be-filtered image block from the first cache, and filters the to-be-filtered image block to obtain a filtered image block; the filtered image blocks are cached to a second cache shared by the first processor and the second processor; and the first processor reads the filtered image block from the second cache and processes the filtered image block, so that the image processing efficiency can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of coding and decoding technology, and in particular to an image processing method, apparatus, device, medium, and program product. Background Art

[0002] Heterogeneous encoders and decoders can fully utilize the powerful computing capabilities of the Graphics Processing Unit (GPU) to execute complex algorithms, such as filtering algorithms. The GPU reads the image block to be filtered and performs the filtering algorithm on it to obtain the filtered image block. The Central Processing Unit (CPU) then retrieves and processes the filtered image block.

[0003] Currently, it is necessary to create a GPU-accessible image input buffer and image output buffer through the Open Computing Language (OpenCL) framework. Based on this, it is first necessary to manually copy the image blocks to be filtered from the image input buffer accessible to the CPU to the image input buffer accessible to the GPU. Then, the GPU can read the image blocks to be filtered from its accessible image input buffer and perform filtering. The filtered image blocks are then cached in the image output buffer accessible to the GPU. At this time, it is necessary to manually copy the filtered image blocks from the image output buffer accessible to the GPU to the image output buffer accessible to the CPU. The CPU reads the filtered image blocks from the buffer and processes them.

[0004] However, this method requires performing two manual image block copying processes, resulting in low image processing efficiency. Summary of the Invention

[0005] The present application provides an image processing method, apparatus, device, medium and program product, thereby improving image processing efficiency.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to a heterogeneous encoder or a heterogeneous decoder, and the heterogeneous encoder or the heterogeneous decoder includes: a first processor and a second processor; wherein the method includes: the first processor obtains an image block to be filtered, and caches the image block to be filtered in a first cache shared by the first processor and the second processor; the second processor reads the image block to be filtered from the first cache, filters the image block to be filtered, and obtains a filtered image block; and caches the filtered image block in a second cache shared by the first processor and the second processor; the first processor reads the filtered image block from the second cache, and processes the filtered image block.

[0007] In a second aspect, an embodiment of the present application provides an image processing method, which is applied to a heterogeneous encoder or a heterogeneous decoder, and the heterogeneous encoder or the heterogeneous decoder includes: a first processor and a second processor; wherein the method includes: the first processor obtains an image block to be filtered, and caches the image block to be filtered in a first cache accessible to the first processor; the first processor obtains a copy instruction for the image block to be filtered, and in response to the copy instruction, copies the image block to be filtered from the first cache to a second cache accessible to the second processor; the second processor reads the image block to be filtered from the second cache, filters the image block to be filtered, and obtains a filtered image block; and caches the filtered image block in a third cache shared by the first processor and the second processor; the first processor reads the filtered image block from the third cache, and processes the filtered image block.

[0008] In a third aspect, an embodiment of the present application provides an image processing method, which is applied to a heterogeneous encoder or a heterogeneous decoder, and the heterogeneous encoder or the heterogeneous decoder includes: a first processor and a second processor; wherein the method includes: the first processor obtains an image block to be filtered, and caches the image block to be filtered in a first cache shared by the first processor and the second processor; the second processor reads the image block to be filtered from the first cache, and filters the image block to be filtered to obtain a filtered image block; and caches the filtered image block in a second cache accessible to the second processor; the second processor obtains a copy instruction for the filtered image block, and in response to the copy instruction, copies the filtered image block from the second cache to a third cache accessible to the first processor; the first processor reads the filtered image block from the third cache, and processes the filtered image block.

[0009] In a fourth aspect, an embodiment of the present application provides an image processing device, comprising: a first processing module and a second processing module; wherein the first processing module is used to obtain an image block to be filtered, and cache the image block to be filtered in a first cache shared by the first processing module and the second processing module; the second processing module is used to read the image block to be filtered from the first cache, and filter the image block to be filtered to obtain a filtered image block; and cache the filtered image block in a second cache shared by the first processing module and the second processing module; the first processing module is also used to read the filtered image block from the second cache and process the filtered image block.

[0010] In a fifth aspect, an embodiment of the present application provides an image processing device, comprising: a first processing module and a second processing module; wherein the first processing module is used to obtain an image block to be filtered, and cache the image block to be filtered in a first cache accessible to the first processing module; the first processing module is also used to obtain a copy instruction for the image block to be filtered, and in response to the copy instruction, copy the image block to be filtered from the first cache to a second cache accessible to the second processing module; the second processing module is used to read the image block to be filtered from the second cache, and filter the image block to be filtered to obtain a filtered image block; and cache the filtered image block in a third cache shared by the first processing module and the second processing module; the first processing module is also used to read the filtered image block from the third cache and process the filtered image block.

[0011] In a sixth aspect, an embodiment of the present application provides an image processing device, comprising: a first processing module and a second processing module; wherein the first processing module is used to obtain an image block to be filtered, and cache the image block to be filtered in a first cache shared by the first processing module and the second processing module; the second processing module is used to read the image block to be filtered from the first cache, and filter the image block to be filtered to obtain a filtered image block; and cache the filtered image block in a second cache accessible to the second processing module; the second processing module is also used to obtain a copy instruction for the filtered image block, and in response to the copy instruction, copy the filtered image block from the second cache to a third cache accessible to the first processing module; the first processing module is also used to read the filtered image block from the third cache and process the filtered image block.

[0012] In the seventh aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the methods as described in the first to third aspects or their respective implementations.

[0013] In an eighth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method as described in the first to third aspects or their respective implementations.

[0014] In a ninth aspect, a computer program product is provided, comprising computer program instructions, which enable a computer to execute the method according to the first to third aspects or their respective implementations.

[0015] In a tenth aspect, a computer program is provided, which enables a computer to execute the method as described in the first to third aspects or their respective implementations.

[0016] Through the technical solution provided by the present application, since the first processor and the second processor share the image input cache and / or the image output cache, there is no need to perform the manual image block copying process twice or only perform the manual image block copying process once, thereby improving the image processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic block diagram of a video encoding and decoding system involved in an embodiment of the present application;

[0018] Figure 2 is a schematic block diagram of a video encoder 200 involved in an embodiment of the present application;

[0019] Figure 3 is a schematic block diagram of a video decoder involved in an embodiment of the present application;

[0020] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0021] Figure 5 A flowchart of another image processing method provided in an embodiment of the present application;

[0022] Figure 6 A flowchart of another image processing method provided in an embodiment of the present application;

[0023] Figure 7 A schematic diagram of an image processing device 700 provided in an embodiment of the present application;

[0024] Figure 8 A schematic diagram of an image processing device 800 provided in an embodiment of the present application;

[0025] Figure 9 A schematic diagram of an image processing device 900 provided in an embodiment of the present application;

[0026] Figure 10 It is a schematic block diagram of an electronic device 1000 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Before introducing the technical solution of this application, the following will explain the relevant knowledge of the technical solution of this application:

[0030] 1. The solution provided in the embodiments of the present application can be applied in the field of coding and decoding technology.

[0031] For example, the technical fields to which the solutions provided in the embodiments of the present application can be applied include but are not limited to: image coding and decoding fields, video coding and decoding fields, hardware video coding and decoding fields, dedicated circuit video coding and decoding fields, and real-time video coding and decoding fields. In addition, the solutions provided in the embodiments of the present application can be combined with the audio and video coding standard (AVS), the second-generation AVS standard (AVS2), or the third-generation AVS standard (AVS3). For example, including but not limited to: H.264 / Audio Video Coding (AVC) standard, H.265 / High Efficiency Video Coding (HEVC) standard, and H.266 / Versatile Video Coding (VVC) standard.

[0032] To facilitate understanding, first combine Figure 1 The video encoding and decoding system involved in the embodiments of the present application is introduced.

[0033] Figure 1 This is a schematic block diagram of a video encoding and decoding system involved in an embodiment of the present application.

[0034] like Figure 1 As shown, the video coding and decoding system 100 includes an encoding device 110 and a decoding device 120 .

[0035] The encoding device 110 is used to encode (which can be understood as compressing) the video data to generate a code stream, and transmit the code stream to the decoding device 120. The decoding device 120 decodes the code stream generated by the encoding device 110 to obtain decoded video data.

[0036] The encoding device 110 can be understood as a device with a video encoding function, and the decoding device 120 can be understood as a device with a video decoding function, that is, the embodiments of the present application include a wider range of devices for the encoding device 110 and the decoding device 120, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, car computers, etc.

[0037] The encoding device 110 may transmit the encoded video data (eg, a code stream) to the decoding device 120 via the channel 130 .

[0038] Channel 130 may include one or more media and / or devices capable of transmitting encoded video data from encoding device 110 to decoding device 120 .

[0039] Channel 130 may include one or more communication media that enable encoding device 110 to transmit encoded video data directly to decoding device 120 in real time. Encoding device 110 may modulate the encoded video data according to a communication standard and transmit the modulated video data to decoding device 120. The communication media may include wireless communication media, such as radio frequency spectrum. The communication media may also include wired communication media, such as one or more physical transmission lines.

[0040] Channel 130 may include a storage medium that can store the video data encoded by encoding device 110. The storage medium includes various locally accessible data storage media, such as optical disks, DVDs, flash memories, etc. In this example, decoding device 120 may obtain the encoded video data from the storage medium.

[0041] Channel 130 may include a storage server that can store the video data encoded by encoding device 110. In this example, decoding device 120 can download the stored encoded video data from the storage server. Alternatively, the storage server can store the encoded video data and transmit the encoded video data to decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

[0042] The encoding device 110 includes a video encoder 112 and an output interface 113 .

[0043] The output interface 113 may include a modulator / demodulator (modem) and / or a transmitter. The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may also be stored on a storage medium or storage server for subsequent reading by the decoding device 120.

[0044] In addition to the video encoder 112 and the input interface 113 , the encoding device 110 may also include a video source 111 .

[0045] Video source 111 may include at least one of a video capture device (e.g., a video camera), a video archive, a video input interface, and a computer graphics system. The video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate video data. Video encoder 112 encodes the video data from video source 111 to generate a bitstream. The video data may include one or more pictures or a sequence of pictures. The bitstream contains encoding information for the picture or picture sequence in the form of a bitstream. The encoding information may include the encoded picture data and associated data. The associated data may include a sequence parameter set (SPS), a picture parameter set (PPS), and other syntax structures. An SPS may contain parameters applicable to one or more sequences. A PPS may contain parameters applicable to one or more pictures. A syntax structure refers to a set of zero or more syntax elements arranged in a specified order within the bitstream.

[0046] The decoding apparatus 120 includes an input interface 121 and a video decoder 122. The input interface 121 may include a receiver and / or a modem.

[0047] The decoding device 120 may include a display device 123 in addition to the input interface 121 and the video decoder 122 .

[0048] The input interface 121 can receive encoded video data via the channel 130. The video decoder 122 is configured to decode the encoded video data to obtain decoded video data, and transmit the decoded video data to the display device 123. The display device 123 displays the decoded video data. The display device 123 can be integrated with the decoding device 120 or external to the decoding device 120. The display device 123 can include various display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

[0049] It should be understood that Figure 1This is only an example of the present application and should not be understood as a display of the present application. In other words, the technical solutions of the embodiments of the present application are not limited to Figure 1 The system framework shown, for example, the technology of the present application can also be applied to single-sided video encoding or single-sided video decoding.

[0050] Figure 2 2 is a schematic block diagram of a video encoder 200 according to an embodiment of the present application.

[0051] It should be understood that the video encoder 200 can be applied to image data in luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in terms of color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0052] The video encoder 200 reads video data and, for each frame of the video data, divides the frame into a number of coding tree units (CTUs). In some examples, a CTB may be referred to as a "tree block," "largest coding unit" (LCU), or "coding tree block" (CTB). Each CTU may be associated with a block of pixels of equal size within the image. Each pixel may correspond to one luminance (luma) sample and two chrominance (chroma) samples. Therefore, each CTU may be associated with one luminance sample block and two chrominance sample blocks. The size of a CTU may be, for example, 128×128, 64×64, 32×32, etc. A CTU may be further divided into a number of coding units (CUs) for encoding. A CU may be a rectangular block or a square block. A CU may be further divided into a prediction unit (PU) and a transform unit (TU), thereby separating encoding, prediction, and transform, making processing more flexible. In an example, a CTU is divided into CUs in a quadtree manner, and a CU is divided into TUs and PUs in a quadtree manner.

[0053] The video encoder and video decoder can support various PU sizes. Assuming that the size of a particular CU is 2N×2N, the video encoder and video decoder can support PU sizes of 2N×2N or N×N for intra-frame prediction, and support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. The video encoder and video decoder can also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0054] like Figure 2 As shown, the video encoder 200 may include: a prediction unit 210, a residual unit 220, a transform / quantization unit 230, an inverse transform / quantization unit 240, a reconstruction unit 250, a loop filter unit 260, a decoded image buffer 270 and an entropy coding unit 280. It should be noted that the video encoder 200 may include more, fewer or different functional components. In this application, the current block may be referred to as the current coding unit (CU) or the current prediction unit (PU), etc. The prediction block may also be referred to as a predicted image block or an image prediction block, and the reconstructed image block may also be referred to as a reconstructed block or an image reconstructed image block.

[0055] The prediction unit 210 includes an inter-frame prediction unit 211 and an intra-frame prediction unit 212. Because there is a strong correlation between adjacent pixels in a video frame, intra-frame prediction is used in video coding and decoding to eliminate spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent frames in a video, inter-frame prediction is used in video coding and decoding to eliminate temporal redundancy between adjacent frames, thereby improving coding efficiency.

[0056] The inter-frame prediction unit 211 can be used for inter-frame prediction. Inter-frame prediction includes motion estimation and motion compensation. It can refer to image information from different frames. Inter-frame prediction uses motion information to find a reference block from a reference frame and generates a prediction block based on the reference block to eliminate temporal redundancy. The frames used for inter-frame prediction can be P-frames and / or B-frames. P-frames refer to forward-predicted frames, while B-frames refer to bidirectionally predicted frames. Inter-frame prediction uses motion information to find a reference block from a reference frame and generates a prediction block based on the reference block. Motion information includes the reference frame list in which the reference frame is located, the reference frame index, and the motion vector. The motion vector can be integer pixel or fractional pixel. If the motion vector is fractional pixel, interpolation filtering is required to generate the required fractional pixel block in the reference frame. Here, the integer pixel or fractional pixel block in the reference frame found based on the motion vector is called a reference block. Some technologies directly use the reference block as the prediction block, while others further process the reference block to generate a prediction block. Further processing the reference block to generate a prediction block can also be understood as using the reference block as the prediction block and then processing the prediction block to generate a new prediction block.

[0057] The intra-frame prediction unit 212 only refers to the information of the same frame image to predict the pixel information in the current code image block to eliminate spatial redundancy. The frame used for intra-frame prediction can be an I frame.

[0058] There are multiple prediction modes for intra-frame prediction. Taking the H series of international digital video coding standards as an example, the H.264 / AVC standard has 8 angular prediction modes and 1 non-angular prediction mode, and H.265 / HEVC has expanded to 33 angular prediction modes and 2 non-angular prediction modes. The intra-frame prediction modes used by HEVC are planar mode (Planar), DC, and 33 angular modes, for a total of 35 prediction modes. The intra-frame modes used by VVC are Planar, DC, and 65 angular modes, for a total of 67 prediction modes. It should be noted that with the increase in angular modes, intra-frame prediction will become more accurate and more in line with the needs of the development of high-definition and ultra-high-definition digital video.

[0059] The residual unit 220 may generate a residual block for the CU based on the pixel blocks of the CU and the prediction blocks of the PUs of the CU. For example, the residual unit 220 may generate the residual block for the CU such that each sample in the residual block has a value equal to the difference between the sample in the pixel blocks of the CU and the corresponding sample in the prediction blocks of the PUs of the CU.

[0060] The transform / quantization unit 230 may quantize the transform coefficients. The transform / quantization unit 230 may quantize the transform coefficients associated with the TUs of a CU based on a quantization parameter (QP) value associated with the CU. The video encoder 200 may adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

[0061] The inverse transform / quantization unit 240 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficients to reconstruct a residual block from the quantized transform coefficients.

[0062] Reconstruction unit 250 may add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by prediction unit 210 to generate a reconstructed image block associated with the TU. By reconstructing the sample blocks of each TU of a CU in this manner, video encoder 200 can reconstruct the pixel blocks of the CU.

[0063] The loop filter unit 260 processes the inverse-transformed and inverse-quantized pixels to compensate for distortion and provide a better reference for subsequent pixel encoding. For example, it can perform a deblocking filter (DF) operation to reduce the blocking artifacts of pixel blocks associated with the CU. In some embodiments, the loop filter unit 260 includes a deblocking filter (DFF) unit and a sample adaptive offset (SAO) / adaptive loop filter (ALF) unit. The DFF unit is used to remove blocking artifacts, while the SAO / ALF unit is used to remove ringing artifacts.

[0064] The decoded image buffer 270 may store the reconstructed pixel blocks. The inter-prediction unit 211 may use a reference image containing the reconstructed pixel blocks to perform inter-prediction on PUs of other images. In addition, the intra-prediction unit 212 may use the reconstructed pixel blocks in the decoded image buffer 270 to perform intra-prediction on other PUs in the same image as the CU.

[0065] The entropy coding unit 280 may receive the quantized transform coefficients from the transform / quantization unit 230. The entropy coding unit 280 may perform one or more entropy coding operations on the quantized transform coefficients to generate entropy-coded data.

[0066] Figure 3 It is a schematic block diagram of a video decoder involved in an embodiment of the present application.

[0067] like Figure 3As shown, the video decoder 300 includes an entropy decoding unit 310, a prediction unit 320, an inverse quantization / transformation unit 330, a reconstruction unit 340, a loop filter unit 350, and a decoded picture buffer 360. It should be noted that the video decoder 300 may include more, fewer, or different functional components.

[0068] The video decoder 300 may receive a bitstream. The entropy decoding unit 310 may parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, the entropy decoding unit 310 may parse the entropy-encoded syntax elements in the bitstream. The prediction unit 320, the inverse quantization / transform unit 330, the reconstruction unit 340, and the loop filter unit 350 may decode the video data based on the syntax elements extracted from the bitstream, thereby generating decoded video data.

[0069] The prediction unit 320 includes an intra prediction unit 322 and an inter prediction unit 321 .

[0070] The intra-prediction unit 322 may perform intra-prediction to generate a prediction block for the PU. The intra-prediction unit 322 may use an intra-prediction mode to generate a prediction block for the PU based on pixel blocks of spatially neighboring PUs. The intra-prediction unit 322 may also determine the intra-prediction mode for the PU based on one or more syntax elements parsed from the codestream.

[0071] The inter-frame prediction unit 321 may construct a first reference picture list (List 0) and a second reference picture list (List 1) based on syntax elements parsed from the codestream. In addition, if a PU is encoded using inter-frame prediction, the entropy decoding unit 310 may parse the motion information of the PU. The inter-frame prediction unit 321 may determine one or more reference blocks for the PU based on the motion information of the PU. The inter-frame prediction unit 321 may generate a prediction block for the PU based on the one or more reference blocks of the PU.

[0072] The inverse quantization / transform unit 330 may inversely quantize (i.e., dequantize) the transform coefficients associated with the TU. The inverse quantization / transform unit 330 may use the QP value associated with the CU of the TU to determine the degree of quantization. After inverse quantizing the transform coefficients, the inverse quantization / transform unit 330 may apply one or more inverse transforms to the inverse quantized transform coefficients to generate a residual block associated with the TU.

[0073] The reconstruction unit 340 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, the reconstruction unit 340 can add samples of the residual block to corresponding samples of the prediction block to reconstruct the pixel block of the CU to obtain a reconstructed image block.

[0074] The loop filtering unit 350 may perform a deblocking filtering operation to reduce blocking artifacts of pixel blocks associated with a CU.

[0075] The video decoder 300 may store the reconstructed image of the CU in the decoded image buffer 360. The video decoder 300 may use the reconstructed image in the decoded image buffer 360 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0076] The basic process of video encoding and decoding is as follows: At the encoder end, a frame of image is divided into blocks. For the current block, the prediction unit 210 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual unit 220 calculates a residual block based on the predicted block and the original block of the current block. This residual block is the difference between the predicted block and the original block of the current block. This residual block is also called residual information. This residual block undergoes transformation and quantization by the transform / quantization unit 230, removing information that is insensitive to the human eye and eliminating visual redundancy. Optionally, the residual block before transformation and quantization by the transform / quantization unit 230 is called a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform / quantization unit 230 is called a frequency residual block or a frequency-domain residual block. The entropy coding unit 280 receives the quantized change coefficients output by the change quantization unit 230 and performs entropy coding on these quantized change coefficients to output a bitstream. For example, the entropy coding unit 280 can eliminate character redundancy based on the target context model and probability information of the binary bitstream.

[0077] At the decoding end, the entropy decoding unit 310 can parse the code stream to obtain the prediction information, quantization coefficient matrix, etc. of the current block. The prediction unit 320 uses intra-frame prediction or inter-frame prediction for the current block based on the prediction information to generate a prediction block for the current block. The inverse quantization / transformation unit 330 uses the quantization coefficient matrix obtained from the code stream to inverse quantize and inverse transform the quantization coefficient matrix to obtain a residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image, and the loop filtering unit 350 performs loop filtering on the reconstructed image based on the image or block to obtain a decoded image. The encoding end also needs to use operations similar to those of the decoder to obtain a decoded image. The decoded image can also be called a reconstructed image, and the reconstructed image can be a subsequent frame, which serves as a reference frame for inter-frame prediction.

[0078] It should be noted that the block division information determined by the encoder, as well as mode information or parameter information such as prediction, transformation, quantization, entropy coding, and loop filtering, etc., are carried in the code stream when necessary. The decoding end determines the same block division information, prediction, transformation, quantization, entropy coding, loop filtering, etc. mode information or parameter information as the encoding end by parsing the code stream and analyzing the existing information, thereby ensuring that the decoded image obtained by the encoder and the decoder are the same. In addition, in an embodiment of the present application, due to the need for parallel processing, the image can be divided into slices, etc., and the slices in the same image can be processed in parallel, that is, there is no data dependency between them. "Frame" is a commonly used term, and it can generally be understood that one frame is an image. Of course, the frame described in this application can also be replaced by an image or a slice, etc.

[0079] It is worth noting that the above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. This application is applicable to the basic process of the video codec under the block-based hybrid coding framework, but is not limited to the framework and process.

[0080] Second, the OpenCL framework is a cross-platform programming framework designed to support parallel computing on various types of processors, such as CPUs, GPUs, and Digital Signal Processors (DSPs). It allows developers to write programs that can run on different hardware platforms. OpenCL provides an extension to the C language that enables developers to write kernel code that can be executed in parallel on multiple processors. These kernels can perform a large number of computing tasks on a variety of devices, thereby achieving high-performance computing. OpenCL is widely used in graphics rendering, scientific computing, machine learning, and other fields, and it enables programs to fully utilize the computing power of modern hardware.

[0081] The Adaptive Loop Filter (ALF) is a post-processing technology used to improve video compression efficiency and quality. It is applied in the decoder's loop to reduce blocking artifacts and noise generated during the video encoding process. ALF improves video quality by smoothing the edges of the decoded image, reducing coding residuals. ALF adaptively adjusts filter parameters based on video content to achieve optimal deblocking and compression performance.

[0082] Fourth, heterogeneous encoders are high-performance encoder solutions based on heterogeneous computing. They leverage different computing architectures, such as CPUs and GPUs, to collaboratively complete encoding tasks, thereby improving encoding efficiency and performance. Heterogeneous encoders can support encoding of multiple input and output formats and offer high performance, flexibility, and scalability.

[0083] A heterogeneous decoder is a device or software solution that leverages different computing platforms or architectures, such as CPUs, GPUs, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs), to collaboratively complete decoding tasks. Similar to heterogeneous encoders, heterogeneous decoders aim to improve decoding efficiency, performance, and flexibility by combining the strengths of different computing platforms.

[0084] The following is an explanation of the technical problems, inventive concepts, and application scenarios to be solved by this application:

[0085] As mentioned above, it is currently necessary to create a GPU-accessible image input cache and image output cache through the OpenCL framework. Based on this, it is first necessary to manually copy the image blocks to be filtered in the image input cache accessible to the CPU to the image input cache accessible to the GPU. Then the GPU can read the image blocks to be filtered from its accessible image input cache and perform filtering processing. The filtered image blocks are then cached in the image output cache accessible to the GPU. At this time, it is necessary to manually copy the filtered image blocks in the image output cache accessible to the GPU to the image output cache accessible to the CPU. The CPU reads the filtered image blocks from the cache and processes them. However, this method requires performing two manual image block copying processes, resulting in low image processing efficiency.

[0086] In order to solve the above technical problems, an embodiment of the present application proposes that the first processor and the second processor share an image input cache and / or an image output cache, thereby eliminating the need to perform the manual image block copying process twice or only performing the manual image block copying process once, thereby improving image processing efficiency.

[0087] The image processing method provided in the embodiments of the present application can be applied to heterogeneous encoders, specifically to the filtering process in heterogeneous encoders, and can also be applied to heterogeneous decoders, specifically to the filtering process in heterogeneous decoders.

[0088] The technical solutions provided in the embodiments of the present application can be applied to video on demand, video live broadcast scenarios, etc., but are not limited thereto.

[0089] The technical solution of this application will be described in detail below:

[0090] Figure 4 This is a flowchart of an image processing method provided in an embodiment of the present application. The method can be applied to a heterogeneous encoder or a heterogeneous decoder. The heterogeneous encoder or the heterogeneous decoder includes: a first processor and a second processor; wherein the processing capability of the second processor may be higher than the processing capability of the first processor, but is not limited thereto; the first processor may be a CPU, and the second processor may be a GPU, but is not limited thereto. Figure 4 As shown, the image processing method may include:

[0091] S410: The first processor obtains an image block to be filtered, and caches the image block to be filtered in a first cache shared by the first processor and the second processor;

[0092] In some implementations, the image block to be filtered may be any of the following, but not limited to: a reconstructed image block, or an image block filtered by at least one filtering method. The at least one filtering method includes, but not limited to, at least one of a deblocking filter, a SAO filter, a constrained directional enhancement filter, and an ALF filter.

[0093] It should be understood that the above-mentioned reconstructed image block refers to an image block that has not been subjected to any filtering. For example, the reconstructed image block may be Figure 2 The reconstructed block generated by the reconstruction unit 250 is shown. Based on this, the subsequent filtering performed by the second processor on the image block to be filtered includes at least one of the following: deblocking filtering, SAO filtering, constrained directional enhancement filtering, and ALF filtering. If the second processor performs at least two filtering operations on the image block to be filtered, the embodiment of the present application does not limit the order of these filtering operations.

[0094] The following is an exemplary description of an image block after being filtered by at least one filtering method:

[0095] For example, the image block filtered by at least one filtering method may be an image block after deblocking filtering. Based on this, the subsequent filtering performed by the second processor on the filtered image block includes at least one of the following: SAO filtering, constrained direction enhancement filtering and ALF filtering. Among them, if the second processor performs at least two filtering on the filtered image block, then the embodiment of the present application does not limit the order of these filtering.

[0096] For example, the image block after being filtered by at least one filtering method may be an image block after undergoing deblocking filtering and SAO filtering. The present embodiment of the application does not restrict the order of deblocking filtering and SAO filtering. Based on this, the subsequent filtering performed by the second processor on the image block to be filtered includes at least one of the following: constrained direction enhancement filtering and ALF filtering. If the second processor performs at least two filtering operations on the image block to be filtered, the present embodiment of the application does not restrict the order of these filtering operations.

[0097] It should be understood that the first cache is used to cache the image block to be filtered, that is, to cache the input image of the second processor. Based on this, the first cache can be understood as an image input cache shared by the first processor and the second processor.

[0098] In some implementations, the first buffer is created when the heterogeneous encoder or the heterogeneous decoder is initialized.

[0099] In some implementations, the first cache is created using a cross-platform coding framework.

[0100] In some implementations, the cross-platform coding framework is an OpenCL framework, an Open Graphics Library (OpenGL) framework, Vulkan, or Metal, etc., but is not limited thereto.

[0101] The OpenGL framework is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics. This API consists of nearly 350 different function calls, capable of drawing everything from simple graphics to complex 3D scenes.

[0102] Vulkan is a cross-platform 2D and 3D graphics API that provides an explicit, streamlined, portable, and extensible API designed for use in embedded systems constrained by processor power, memory limitations, and power consumption, as well as in more complex systems with fewer constraints.

[0103] Metal is a low-level graphics programming interface. By using related APIs, developers can directly operate the GPU. The Metal framework supports GPU hardware acceleration, advanced 3D graphics rendering, and large-scale parallel computing. It provides an advanced yet streamlined API to ensure fine-grained control of the framework and supports low-level control over the organization structure, program processing, graphics rendering, operation instructions, and the management of instruction-related data resources. Its core goal is to minimize CPU overhead and offload the majority of runtime load to the GPU.

[0104] S420: The second processor reads the image block to be filtered from the first cache, filters the image block to be filtered, and obtains a filtered image block; and caches the filtered image block in a second cache shared by the first processor and the second processor;

[0105] In some implementations, the second processor may perform at least one of deblocking filtering, SAO filtering, constrained direction enhancement filtering, and ALF filtering on the image block to be filtered to obtain the filtered image block.

[0106] It should be understood that the second cache is used to cache the filtered image blocks, that is, to cache the output image of the second processor. Based on this, the second cache can be understood as an image output cache shared by the first processor and the second processor.

[0107] In some implementations, the second buffer is created when the heterogeneous encoder or the heterogeneous decoder is initialized.

[0108] In some implementations, the second cache is created using a cross-platform coding framework.

[0109] In some implementations, the cross-platform coding framework is an OpenCL framework, an OpenGL framework, Vulkan, or Metal, etc., but is not limited thereto.

[0110] It should be understood that the explanations about the OpenCL framework, OpenGL framework, Vulkan and Metal can be referred to above, and the embodiments of the present application will not be repeated here.

[0111] S430: The first processor reads the filtered image block from the second cache and processes the filtered image block.

[0112] In some implementations, if the image processing method is performed by a heterogeneous encoder, the first processor processes the filtered image block, including: the first processor uses the filtered image block as a reference image block, and uses a reference image containing the reference image block to perform inter-frame prediction on a PU of another image. Alternatively, the first processor uses the reference image block to perform intra-frame prediction on another PU in the same image as the CU.

[0113] In some implementations, if the image processing method is performed by a heterogeneous decoder, the first processor processes the filtered image block, including: the first processor may use the filtered image block as a reference image for subsequent prediction, or transmit the filtered image block to a display device for presentation.

[0114] An embodiment of the present application provides an image processing method, comprising: a first processor obtaining an image block to be filtered and caching the image block to be filtered in a first cache shared by the first processor and a second processor; a second processor reading the image block to be filtered from the first cache and filtering the image block to obtain a filtered image block; and caching the filtered image block in a second cache shared by the first processor and the second processor; and the first processor reading the filtered image block from the second cache and processing the filtered image block. Because the first and second processors share the first and second caches, there is no need to manually copy the image block twice, thereby improving image processing efficiency.

[0115] In addition, the image processing method provided in the embodiment of the present application involves two caches, which is two less than the related art, thereby reducing memory consumption.

[0116] Figure 5 This is a flowchart of another image processing method provided in an embodiment of the present application. The method can be applied to a heterogeneous encoder or a heterogeneous decoder. The heterogeneous encoder or the heterogeneous decoder includes: a first processor and a second processor; wherein the processing capability of the second processor can be higher than the processing capability of the first processor, but is not limited thereto. The first processor can be a CPU, and the second processor can be a GPU, but is not limited thereto. Figure 5 As shown, the image processing method may include:

[0117] S510: The first processor obtains an image block to be filtered, and caches the image block to be filtered in a first cache accessible to the first processor;

[0118] In some implementations, the image block to be filtered may be any of the following, but not limited to: a reconstructed image block, or an image block filtered by at least one filtering method. The at least one filtering method includes, but not limited to, at least one of a deblocking filter, a SAO filter, a constrained directional enhancement filter, and an ALF filter.

[0119] It should be understood that the explanation of the reconstructed image block and the image block after filtering by at least one filtering method can be referred to Figure 4 The corresponding method embodiments will not be described in detail in the embodiments of this application.

[0120] It should be understood that the first cache is used to cache the image blocks to be filtered, that is, to cache the input image of the second processor. Based on this, the first cache can be understood as an image input cache accessible to the first processor.

[0121] S520: The first processor obtains a copy instruction for the image block to be filtered, and in response to the copy instruction, copies the image block to be filtered from the first cache to a second cache accessible to the second processor;

[0122] It should be understood that the copy instruction is used to copy the image block to be filtered from the first cache to the second cache accessible to the second processor.

[0123] It should be understood that the second cache is used to cache the image blocks to be filtered, that is, to cache the input image of the second processor. Based on this, the second cache can be understood as an image input cache accessible to the second processor.

[0124] In some implementations, the second buffer is created when the heterogeneous encoder or the heterogeneous decoder is initialized.

[0125] In some implementations, the second cache is created using a cross-platform coding framework.

[0126] It should be understood that the explanation of the cross-platform coding framework can be found in Figure 4 The corresponding method embodiments will not be described in detail in the embodiments of this application.

[0127] S530: The second processor reads the image block to be filtered from the second cache, filters the image block to be filtered, and obtains a filtered image block; and caches the filtered image block in a third cache shared by the first processor and the second processor;

[0128] In some implementations, the second processor may perform at least one of deblocking filtering, SAO filtering, constrained direction enhancement filtering, and ALF filtering on the image block to be filtered to obtain the filtered image block.

[0129] It should be understood that the third cache is used to cache the filtered image blocks, that is, to cache the output image of the second processor. Based on this, the third cache can be understood as an image output cache shared by the first processor and the second processor.

[0130] In some implementations, the third buffer is created when the heterogeneous encoder or the heterogeneous decoder is initialized.

[0131] In some implementations, the third cache is created using a cross-platform encoding framework.

[0132] It should be understood that the explanation of the cross-platform coding framework can be found in Figure 4 The corresponding method embodiments will not be described in detail in the embodiments of this application.

[0133] S540: The first processor reads the filtered image block from the third cache and processes the filtered image block.

[0134] In some implementations, if the image processing method is performed by a heterogeneous encoder, the first processor processes the filtered image block, including: the first processor uses the filtered image block as a reference image block, and uses a reference image containing the reference image block to perform inter-frame prediction on a PU of another image. Alternatively, the first processor uses the reference image block to perform intra-frame prediction on another PU in the same image as the CU.

[0135] In some implementations, if the image processing method is performed by a heterogeneous decoder, the first processor processes the filtered image block, including: the first processor may use the filtered image block as a reference image for subsequent prediction, or transmit the filtered image block to a display device for presentation.

[0136] An embodiment of the present application provides an image processing method, comprising: a first processor obtaining an image block to be filtered and caching the image block to be filtered in a first cache accessible to the first processor; the first processor obtaining a copy instruction for the image block to be filtered and, in response to the copy instruction, copying the image block to be filtered from the first cache to a second cache accessible to a second processor; the second processor reading the image block to be filtered from the second cache and filtering the image block to obtain a filtered image block; and caching the filtered image block in a third cache shared by the first and second processors; and the first processor reading the filtered image block from the third cache and processing the filtered image block. Because the first and second processors share the third cache, the manual image block copying process only needs to be performed once, thereby improving image processing efficiency.

[0137] In addition, the image processing method provided in the embodiment of the present application involves three caches, which is one less cache than the related art, thereby reducing memory consumption.

[0138] Figure 6 This is a flowchart of another image processing method provided in an embodiment of the present application. The method can be applied to a heterogeneous encoder or a heterogeneous decoder. The heterogeneous encoder or the heterogeneous decoder includes: a first processor and a second processor; wherein the processing capability of the second processor can be higher than the processing capability of the first processor, but is not limited thereto. The first processor can be a CPU, and the second processor can be a GPU, but is not limited thereto. Figure 6 As shown, the image processing method may include:

[0139] S610: The first processor obtains an image block to be filtered, and caches the image block to be filtered in a first cache shared by the first processor and the second processor;

[0140] In some implementations, the image block to be filtered may be any of the following, but not limited to: a reconstructed image block, or an image block filtered by at least one filtering method. The at least one filtering method includes, but not limited to, at least one of a deblocking filter, a SAO filter, a constrained directional enhancement filter, and an ALF filter.

[0141] It should be understood that the explanation of the reconstructed image block and the image block after filtering by at least one filtering method can be referred to Figure 4 The corresponding method embodiments will not be described in detail in the embodiments of this application.

[0142] It should be understood that the first cache is used to cache the image block to be filtered, that is, to cache the input image of the second processor. Based on this, the first cache can be understood as an image input cache shared by the first processor and the second processor.

[0143] In some implementations, the first buffer is created when the heterogeneous encoder or the heterogeneous decoder is initialized.

[0144] In some implementations, the first cache is created using a cross-platform coding framework.

[0145] It should be understood that the explanation of the cross-platform coding framework can be found in Figure 4 The corresponding method embodiments will not be described in detail in the embodiments of this application.

[0146] S620: The second processor reads the image block to be filtered from the first cache, filters the image block to be filtered, and obtains a filtered image block; and caches the filtered image block in a second cache accessible to the second processor;

[0147] In some implementations, the second processor may perform at least one of deblocking filtering, SAO filtering, constrained direction enhancement filtering, and ALF filtering on the image block to be filtered to obtain the filtered image block.

[0148] It should be understood that the second cache is used to cache the filtered image blocks, that is, to cache the output image of the second processor. Based on this, the second cache can be understood as an image output cache accessible to the second processor.

[0149] In some implementations, the second buffer is created when the heterogeneous encoder or the heterogeneous decoder is initialized.

[0150] In some implementations, the second cache is created using a cross-platform coding framework.

[0151] It should be understood that the explanation of the cross-platform coding framework can be found in Figure 4The corresponding method embodiments will not be described in detail in the embodiments of this application.

[0152] S630: The second processor obtains a copy instruction for the filtered image block, and in response to the copy instruction, copies the filtered image block from the second cache to a third cache accessible to the first processor;

[0153] It should be understood that the copy instruction is used to copy the filtered image block from the second cache to the third cache accessible to the first processor.

[0154] It should be understood that the third cache is used to cache the filtered image blocks, that is, to cache the output image of the second processor. Based on this, the third cache can be understood as an image output cache accessible to the first processor.

[0155] S640: The first processor reads the filtered image block from the third cache and processes the filtered image block.

[0156] In some implementations, if the image processing method is performed by a heterogeneous encoder, the first processor processes the filtered image block, including: the first processor uses the filtered image block as a reference image block, and uses a reference image containing the reference image block to perform inter-frame prediction on a PU of another image. Alternatively, the first processor uses the reference image block to perform intra-frame prediction on another PU in the same image as the CU.

[0157] In some implementations, if the image processing method is performed by a heterogeneous decoder, the first processor processes the filtered image block, including: the first processor may use the filtered image block as a reference image for subsequent prediction, or transmit the filtered image block to a display device for presentation.

[0158] An embodiment of the present application provides an image processing method, comprising: a first processor obtaining an image block to be filtered and caching the image block to be filtered in a first cache shared by the first processor and a second processor; a second processor reading the image block to be filtered from the first cache and filtering the image block to obtain a filtered image block; and caching the filtered image block in a second cache accessible to the second processor; the second processor obtaining a copy instruction for the filtered image block and, in response to the copy instruction, copying the filtered image block from the second cache to a third cache accessible to the first processor; and the first processor reading the filtered image block from the third cache and processing the filtered image block. Because the first and second processors share the first cache, the manual image block copying process only needs to be performed once, thereby improving image processing efficiency.

[0159] In addition, the image processing method provided in the embodiment of the present application involves three caches, which is one less cache than the related art, thereby reducing memory consumption.

[0160] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, a variety of simple modifications can be made to the technical solution of the present application, and these simple modifications all fall within the scope of protection of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, and as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0161] It should also be understood that in the various method embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0162] The above describes the method provided in the embodiment of the present application. The following describes the image processing device provided in the embodiment of the present application.

[0163] Figure 7 A schematic diagram of an image processing device 700 provided in an embodiment of the present application is shown as follows: Figure 7 As shown, the image processing device 700 includes: a first processing module 710 and a second processing module 720. The first processing module 710 is used to obtain an image block to be filtered and cache the image block to be filtered in a first cache shared by the first processing module 710 and the second processing module 720; the second processing module 720 is used to read the image block to be filtered from the first cache, filter the image block to be filtered, obtain a filtered image block, and cache the filtered image block in a second cache shared by the first processing module 710 and the second processing module 720; the first processing module 710 is further used to read the filtered image block from the second cache and process the filtered image block.

[0164] In some implementations, the second processing module 720 is specifically configured to perform at least one of deblocking filtering, sample adaptive compensation, constrained direction enhancement filtering, and adaptive loop filtering on the image block to be filtered to obtain a filtered image block.

[0165] In some implementations, at least one of the first cache and the second cache is created when the heterogeneous encoder or the heterogeneous decoder is initialized.

[0166] In some implementations, at least one of the first cache and the second cache is created using a cross-platform coding framework.

[0167] In some implementations, the cross-platform coding framework is an OpenCL framework.

[0168] In some implementations, the first processing module 710 is a CPU, and the second processing module 720 is a GPU.

[0169] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 7 The apparatus 700 shown may perform Figure 4 The corresponding method embodiment, and the aforementioned and other operations and / or functions of each module in the device 700 are respectively to achieve Figure 4 For the sake of brevity, the corresponding processes in each method are not repeated here.

[0170] The above describes the device 700 of the embodiment of the present application from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.

[0171] Figure 8 A schematic diagram of an image processing device 800 provided in an embodiment of the present application is shown as follows: Figure 8As shown, the image processing device 800 includes: a first processing module 810 and a second processing module 820. The first processing module 810 is used to obtain an image block to be filtered and cache the image block to be filtered in a first cache accessible to the first processing module 810; the first processing module 810 is also used to obtain a copy instruction for the image block to be filtered and, in response to the copy instruction, copy the image block to be filtered from the first cache to a second cache accessible to the second processing module 820; the second processing module 820 is used to read the image block to be filtered from the second cache, filter the image block to be filtered, and obtain a filtered image block; and cache the filtered image block in a third cache shared by the first processing module 810 and the second processing module 820; the first processing module 810 is also used to read the filtered image block from the third cache and process the filtered image block.

[0172] In some implementations, the second processing module 820 is specifically configured to perform at least one of deblocking filtering, sample adaptive compensation, constrained direction enhancement filtering, and adaptive loop filtering on the image block to be filtered to obtain a filtered image block.

[0173] In some implementations, at least one of the second buffer and the third buffer is created when the heterogeneous encoder or the heterogeneous decoder is initialized.

[0174] In some implementations, at least one of the second cache and the third cache is created using a cross-platform coding framework.

[0175] In some implementations, the cross-platform coding framework is an OpenCL framework.

[0176] In some implementations, the first processing module 810 is a CPU, and the second processing module 820 is a GPU.

[0177] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 8 The apparatus 800 shown may perform Figure 5 The corresponding method embodiment, and the aforementioned and other operations and / or functions of each module in the device 800 are respectively to achieve Figure 5 For the sake of brevity, the corresponding processes in each method are not repeated here.

[0178] The above describes the device 800 of the embodiment of the present application from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.

[0179] Figure 9 A schematic diagram of an image processing device 900 provided in an embodiment of the present application is shown as follows: Figure 9 As shown, the image processing device 900 includes: a first processing module 910 and a second processing module 920. The first processing module 910 is configured to obtain an image block to be filtered and cache the image block to be filtered in a first cache shared by the first processing module 910 and the second processing module 920; the second processing module 920 is configured to read the image block to be filtered from the first cache, filter the image block to be filtered, obtain a filtered image block, and cache the filtered image block in a second cache accessible to the second processing module 920; the second processing module 920 is further configured to obtain a copy instruction for the filtered image block and, in response to the copy instruction, copy the filtered image block from the second cache to a third cache accessible to the first processing module 910; the first processing module 910 is further configured to read the filtered image block from the third cache and process the filtered image block.

[0180] In some implementations, the second processing module 920 is specifically configured to perform at least one of deblocking filtering, sample adaptive compensation, constrained direction enhancement filtering, and adaptive loop filtering on the image block to be filtered to obtain a filtered image block.

[0181] In some implementations, at least one of the second buffer and the third buffer is created when the heterogeneous encoder or the heterogeneous decoder is initialized.

[0182] In some implementations, at least one of the second cache and the third cache is created using a cross-platform coding framework.

[0183] In some implementations, the cross-platform coding framework is an OpenCL framework.

[0184] In some implementations, the first processing module 910 is a CPU, and the second processing module 920 is, in some implementations, a GPU.

[0185] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 9 The apparatus 900 shown may perform Figure 6 The corresponding method embodiment, and the aforementioned and other operations and / or functions of each module in the device 900 are respectively to achieve Figure 6 For the sake of brevity, the corresponding processes in each method are not repeated here.

[0186] The above describes the device 900 of the embodiment of the present application from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.

[0187] Figure 10 1 is a schematic block diagram of an electronic device 1000 provided in an embodiment of the present application. The electronic device 1000 may be a heterogeneous encoder or a heterogeneous decoder.

[0188] like Figure 10 As shown, the electronic device 1000 may include:

[0189] Memory 1010, first processor 1020, and second processor 1030. The memory 1010 is used to store computer programs and transmit corresponding program codes to the first processor 1020 and the second processor 1030. In other words, the first processor 1020 and the second processor 1030 can call and run the corresponding computer program from the memory 1010 to implement the corresponding method steps in the embodiments of the present application.

[0190] For example, the first processor 1020 and the second processor 1030 may be configured to execute the above method embodiments according to instructions in the computer program.

[0191] In some embodiments of the present application, the first processor 1020 may be a CPU, and the second processor 1030 may be a GPU, but is not limited thereto.

[0192] In some embodiments of the present application, the memory 1010 includes but is not limited to:

[0193] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0194] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 1010 and executed by the first processor 1020 and the second processor 1030 to implement the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0195] like Figure 10 As shown, the electronic device may further include:

[0196] The transceiver 1040 can be connected to the first processor 1020 , the second processor 1030 , or the memory 1010 .

[0197] The first processor 1020 or the second processor 1030 may control the transceiver 1040 to communicate with other devices. Specifically, it may send information or data to other devices or receive information or data sent by other devices. The transceiver 1040 may include a transmitter and a receiver. The transceiver 1040 may further include an antenna, and the number of antennas may be one or more.

[0198] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0199] The present application also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment. In other words, the present application also provides a computer program product containing instructions, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment.

[0200] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0201] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0203] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the present embodiment. For example, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module.

[0204] The above is only a specific embodiment of the present application, but the scope of protection of the present application is not limited to this. Any changes or substitutions that can be easily conceived by any person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: The method is applied to a heterogeneous encoder or a heterogeneous decoder, wherein the heterogeneous encoder or the heterogeneous decoder includes: a first processor and a second processor; wherein the method includes: The first processor obtains the image block to be filtered, and caches the image block to be filtered in a first cache shared by the first processor and the second processor; The second processor reads the image block to be filtered from the first cache, filters the image block to be filtered, and obtains a filtered image block; and caches the filtered image block in a second cache shared by the first processor and the second processor; The first processor reads the filtered image block from the second cache and processes the filtered image block.

2. The method according to claim 1, characterized in that The filtering the image block to be filtered to obtain a filtered image block includes: The second processor performs at least one of deblocking filtering, sample adaptive compensation, constrained direction enhancement filtering, and adaptive loop filtering on the image block to be filtered to obtain the filtered image block.

3. The method according to claim 1 or 2, characterized in that At least one of the first cache and the second cache is created when the heterogeneous encoder or the heterogeneous decoder is initialized.

4. The method according to claim 1 or 2, characterized in that At least one of the first cache and the second cache is created using a cross-platform coding framework.

5. The method according to claim 4, characterized in that The cross-platform coding framework is the Open Computing Language OpenCL framework.

6. The method according to claim 1 or 2, characterized in that The first processor is a central processing unit (CPU), and the second processor is a graphics processing unit (GPU).

7. An image processing method, characterized in that: The method is applied to a heterogeneous encoder or a heterogeneous decoder, wherein the heterogeneous encoder or the heterogeneous decoder includes: a first processor and a second processor; wherein the method includes: The first processor obtains the image block to be filtered, and caches the image block to be filtered in a first cache accessible to the first processor; The first processor obtains a copy instruction for the image block to be filtered, and in response to the copy instruction, copies the image block to be filtered from the first cache to a second cache accessible to the second processor; The second processor reads the image block to be filtered from the second cache, filters the image block to be filtered, and obtains a filtered image block; and caches the filtered image block in a third cache shared by the first processor and the second processor; The first processor reads the filtered image block from the third cache and processes the filtered image block.

8. An image processing method, characterized in that: The method is applied to a heterogeneous encoder or a heterogeneous decoder, wherein the heterogeneous encoder or the heterogeneous decoder includes: a first processor and a second processor; wherein the method includes: The first processor obtains the image block to be filtered, and caches the image block to be filtered in a first cache shared by the first processor and the second processor; The second processor reads the image block to be filtered from the first cache, filters the image block to be filtered, and obtains a filtered image block; and caches the filtered image block in a second cache accessible to the second processor; The second processor obtains a copy instruction for the filtered image block, and in response to the copy instruction, copies the filtered image block from the second cache to a third cache accessible to the first processor; The first processor reads the filtered image block from the third cache and processes the filtered image block.

9. An image processing device, characterized in that: include: a first processing module and a second processing module; The first processing module is used to obtain the image block to be filtered, and cache the image block to be filtered in a first cache shared by the first processing module and the second processing module; The second processing module is configured to read the image block to be filtered from the first cache, filter the image block to be filtered, and obtain a filtered image block; and cache the filtered image block in a second cache shared by the first processing module and the second processing module; The first processing module is further configured to read the filtered image block from the second cache and process the filtered image block.

10. An image processing device, characterized in that: include: a first processing module and a second processing module; The first processing module is used to obtain the image block to be filtered, and cache the image block to be filtered in a first cache accessible to the first processing module; The first processing module is further configured to obtain a copy instruction for the image block to be filtered, and in response to the copy instruction, copy the image block to be filtered from the first cache to a second cache accessible to the second processing module; The second processing module is configured to read the image block to be filtered from the second cache, filter the image block to be filtered, and obtain a filtered image block; and cache the filtered image block in a third cache shared by the first processing module and the second processing module; The first processing module is further configured to read the filtered image block from the third cache and process the filtered image block.

11. An image processing device, characterized in that: include: a first processing module and a second processing module; The first processing module is used to obtain the image block to be filtered, and cache the image block to be filtered in a first cache shared by the first processing module and the second processing module; The second processing module is configured to read the image block to be filtered from the first cache, filter the image block to be filtered, and obtain a filtered image block; and cache the filtered image block in a second cache accessible to the second processing module; The second processing module is further configured to obtain a copy instruction for the filtered image block, and in response to the copy instruction, copy the filtered image block from the second cache to a third cache accessible to the first processing module; The first processing module is further configured to read the filtered image block from the third cache and process the filtered image block.

12. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 8.

13. A computer-readable storage medium, characterized in that Used to store a computer program, which, when running on a computer, causes the computer to perform the method according to any one of claims 1 to 8.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images