Image encoding, decoding method and apparatus, codec

Abstract

An image encoding and decoding method and apparatus, and an encoder and decoder, are disclosed. The method includes: acquiring a visual saliency heatmap of an image in the current frame, and filtering the current frame image using the visual saliency heatmap to obtain a target image; acquiring a motion estimation vector and a target prediction image of the next frame input image using the target image and the next frame input image; and encoding the difference image between the next frame input image and the target prediction image and the motion estimation vector. This improves the accuracy of acquiring the target prediction image and also improves the encoding accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to an image encoding method, encoding device, decoding amplification, decoding device, codec, computer-readable storage medium, and electronic device. Background Technology

[0002] In the video compression coding framework, the loop filtering module and the motion compensation module are key modules for improving compression efficiency and video reconstruction quality.

[0003] Currently, traditional loop filters and motion compensation schemes are based on macroblock design. When performing calculations, it is necessary to perform complex calculations on the relationship between each macroblock and other macroblocks, and to consider more local image information, which limits the improvement in the quality of video compression and reconstruction.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] According to a first aspect of this disclosure, an image encoding method is provided, comprising:

[0006] Obtain the visual saliency heatmap of the current frame image, and use the visual saliency heatmap of the current frame image to filter the image to be processed to obtain the target image;

[0007] The motion estimation vector and the target prediction image of the next frame input image are obtained using the target image and the next frame input image;

[0008] The difference image and motion estimation vector between the next frame input image and the target prediction image are encoded.

[0009] According to a second aspect of this disclosure, an image decoding method is provided, comprising:

[0010] Obtain the reference prediction image for the current frame;

[0011] The decoded difference image is obtained by decoding the encoded data of the difference image between the input image and the reference prediction image of the current frame.

[0012] The image to be processed is obtained based on the decoded differential image and the reference prediction image;

[0013] Obtain the saliency heatmap of the image to be processed in the current frame, and use the visual saliency heatmap of the image to be processed in the current frame to filter the image to be processed to obtain the output image of the current frame.

[0014] A third aspect of this disclosure provides an encoding apparatus, comprising:

[0015] The filtering module is used to obtain the visual saliency heatmap of the current frame image, and use the visual saliency heatmap of the current frame image to filter the image to be processed to obtain the target image.

[0016] The prediction module is used to obtain the motion estimation vector and the target prediction image of the next frame input image using the target image and the next frame input image;

[0017] The encoding module is used to encode the difference image and motion estimation vector between the next frame input image and the target prediction image.

[0018] A fourth aspect of this disclosure provides a decoding apparatus, comprising:

[0019] The acquisition module is used to acquire the reference prediction image for the current frame;

[0020] The decoding module is used to decode the encoded data of the difference image between the input image and the reference prediction image of the current frame to obtain the decoded difference image;

[0021] The generation module is used to obtain the image to be processed based on the decoded differential image and the reference prediction image;

[0022] The filtering module is used to obtain the saliency heatmap of the image to be processed in the current frame, and to use the visual saliency heatmap of the image to be processed in the current frame to filter the image to be processed to obtain the output image of the current frame.

[0023] A fifth aspect of this disclosure provides a codec, comprising:

[0024] The decoding module is used to decode the encoded data of the difference image between the input image of the current frame and the reference prediction image of the current frame to obtain the decoded difference image;

[0025] The generation module is used to obtain the image to be processed based on the decoded differential image and the reference prediction image;

[0026] The filtering module is used to obtain the visual saliency heatmap of the image to be processed in the current frame, and to use the visual saliency heatmap of the image to be processed in the current frame to filter the image to be processed to obtain the target image.

[0027] The prediction module is used to obtain the motion estimation vector and the target prediction image of the next frame input image using the target image and the next frame input image;

[0028] The encoding module is used to encode the difference image and motion estimation vector between the next frame input image and the target prediction image.

[0029] A sixth aspect of this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0030] According to a seventh aspect of this disclosure, an electronic device is provided, comprising:

[0031] The image encoding apparatus, the image decoding apparatus, or the image codec described in any of the preceding claims. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0033] Figure 1 A schematic diagram of an exemplary system architecture to which embodiments of the present disclosure may be applied is shown;

[0034] Figure 2 A schematic diagram of an electronic device to which embodiments of the present disclosure may be applied is shown;

[0035] Figure 3 A flowchart illustrating an image encoding method in an exemplary embodiment of the present disclosure is shown schematically.

[0036] Figure 4 This schematically illustrates a data flow diagram for image filtering of the current frame in an exemplary embodiment of the present disclosure;

[0037] Figure 5 This schematically illustrates a flowchart of acquiring an image to be processed according to an exemplary embodiment of the present disclosure;

[0038] Figure 6 This schematically illustrates a data flow diagram for motion estimation in an exemplary embodiment of the present disclosure;

[0039] Figure 7 A flowchart illustrating an image decoding method according to an exemplary embodiment of the present disclosure is shown schematically.

[0040] Figure 8 This schematic diagram illustrates an image encoding apparatus according to an exemplary embodiment of the present disclosure;

[0041] Figure 9 This schematic diagram illustrates a specific structure of an image encoding apparatus according to an exemplary embodiment of the present disclosure;

[0042] Figure 10 This schematic diagram illustrates an image decoding apparatus according to an exemplary embodiment of the present disclosure;

[0043] Figure 11 This schematic diagram illustrates a specific structure of an image decoding apparatus according to an exemplary embodiment of the present disclosure;

[0044] Figure 12 The illustration schematically shows a diagram of an image codec according to an exemplary embodiment of the present disclosure;

[0045] Figure 13 The schematic diagram illustrates a specific structure of an image codec according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0046] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0047] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0048] Figure 1 A schematic diagram of a system architecture for an exemplary application environment in which an encoding method, encoding apparatus, decoding amplification, and decoding apparatus according to embodiments of the present disclosure can be applied is shown.

[0049] like Figure 1 As shown, system architecture 100 may include one or more of terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables. Terminal devices 101, 102, and 103 may be various electronic devices with encoding and / or decoding functions, including but not limited to desktop computers, portable computers, smartphones, and tablets. It should be understood that... Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0050] The encoding and / or decoding methods provided in this embodiment are generally executed by terminal devices 101, 102, and 103, and correspondingly, encoding and / or decoding devices are generally disposed in terminal devices 101, 102, and 103. However, it is readily understood by those skilled in the art that the encoding and / or decoding methods provided in this embodiment can also be executed by server 105, and correspondingly, encoding and / or decoding devices can also be disposed in server 105. This exemplary embodiment does not impose any special limitations on this. For example, in one exemplary embodiment, a user may use terminal devices 101, 102, and 103 to encode an image and upload the encoded data to server 105. The server then decodes the encoded data using the decoding method provided in this embodiment. Alternatively, the server may encode an image using the encoding method provided in this embodiment and transmit it to terminal devices 101, 102, and 103, and terminal devices 101, 102, and 103 decode the encoded data using the decoding method provided in this embodiment.

[0051] Exemplary embodiments of this disclosure provide an electronic device for implementing an encoding method or a decoding method, which may be... Figure 1 The terminal devices 101, 102, 103, or server 105 are included. The electronic device includes at least a processor and a memory, the memory being used to store executable instructions of the processor.

[0052] The following is based on Figure 2 Taking a mobile terminal 200 as an example, the construction of an electronic device will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 2 The structure shown can also be applied to fixed-type devices. In other embodiments, the mobile terminal 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The components illustrated can be implemented in hardware, software, or a combination of software and hardware. The interface connections between the components are only schematic and do not constitute a limitation on the structure of the mobile terminal 200. In other embodiments, the mobile terminal 200 may also adopt a similar design to... Figure 2 Different interface connection methods, or combinations of multiple interface connection methods.

[0053] like Figure 2As shown, the mobile terminal 200 may specifically include: a processor 210, internal memory 221, external memory interface 222, Universal Serial Bus (USB) interface 230, charging management module 240, power management module 241, battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, speaker 271, receiver 272, microphone 273, headphone jack 274, sensor module 280, display screen 290, camera module 291, indicator 292, motor 293, buttons 294, and subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, etc.

[0054] Processor 210 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0055] NPU stands for Neural Network (NN) computing processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in mobile terminals, such as image recognition, facial recognition, speech recognition, and text understanding.

[0056] The processor 210 includes a memory. The memory can store instructions for implementing six modular functions: detection instructions, link instructions, information management instructions, analysis instructions, data transfer instructions, and notification instructions, and their execution is controlled by the processor 210.

[0057] The charging management module 240 receives charging input from the charger. The power management module 241 connects the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, internal memory 221, display screen 290, camera module 291, and wireless communication module 260, etc.

[0058] The wireless communication function of the mobile terminal 200 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals; mobile communication module 250 can provide wireless communication solutions including 2G / 3G / 4G / 5G for use on the mobile terminal 200; modem processor can include modulator and demodulator; wireless communication module 260 can provide wireless communication solutions including Wireless Local Area Networks (WLAN) (such as Wireless Fidelity (Wi-Fi) networks) and Bluetooth (BT) for use on the mobile terminal 200. In some embodiments, antenna 1 of the mobile terminal 200 is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, enabling the mobile terminal 200 to communicate with networks and other devices via wireless communication technology.

[0059] The mobile terminal 200 implements display functions through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0060] The mobile terminal 200 can perform shooting functions through an ISP, a camera module 291, a video codec, a GPU, a display screen 290, and an application processor. The ISP processes data fed back from the camera module 291; the camera module 291 captures still images or videos; the digital signal processor processes digital signals, including digital image signals and other digital signals; the video codec compresses or decompresses digital video, and the mobile terminal 200 can support one or more video codecs.

[0061] The external storage interface 222 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the mobile terminal 200. The external storage card communicates with the processor 210 through the external storage interface 222 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0062] Internal memory 221 can be used to store executable program code, including instructions. Internal memory 221 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.). The data storage area may store data created during the use of the mobile terminal 200 (such as audio data, phonebook, etc.). Furthermore, internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, Universal Flash Storage (UFS), etc. Processor 210 executes various functional applications and data processing of the mobile terminal 200 by running instructions stored in internal memory 221 and / or instructions stored in memory located in the processor.

[0063] The mobile terminal 200 can implement audio functions such as music playback and recording through an audio module 270, a speaker 271, a receiver 272, a microphone 273, a headphone jack 274, and an application processor.

[0064] The depth sensor 2801 is used to acquire depth information of the scene. In some embodiments, the depth sensor may be disposed on the camera module 291.

[0065] The pressure sensor 2802 is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 2802 may be disposed on the display screen 290. There are many types of pressure sensors 2802, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0066] The gyroscope sensor 2803 can be used to determine the motion attitude of the mobile terminal 200. In some embodiments, the gyroscope sensor 2803 can determine the angular velocity of the mobile terminal 200 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 2803 can be used for image stabilization, navigation, motion-sensing game scenes, etc.

[0067] In addition, other sensors with different functions can be set in the sensor module 280 according to actual needs, such as barometric pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor, bone conduction sensor, etc.

[0068] The mobile terminal 200 may also include other devices that provide auxiliary functions. For example, buttons 294 may include a power button, volume buttons, etc., allowing users to input key signals related to user settings and function control of the mobile terminal 200. Other examples include indicators 292, motors 293, and SIM card interfaces 295.

[0069] The encoding and decoding methods and apparatus of the exemplary embodiments of this disclosure will be described in detail below.

[0070] Figure 3 The flowchart of an encoding method in this exemplary embodiment is shown, including the following steps.

[0071] Step S310: Obtain the visual saliency heatmap of the current frame image, and use the visual saliency heatmap of the current frame image to filter the previous frame image to obtain the target image;

[0072] Step S320: Obtain the motion estimation vector and the target prediction image of the next frame input image using the target image and the next frame input image;

[0073] Step S330: Encode the difference image and motion estimation vector between the next frame input image and the target prediction image.

[0074] In this exemplary embodiment, filtering the image using the visual saliency heatmap of the current frame improves the display effect of the target image, thereby enhancing the accuracy of the target prediction image and improving coding accuracy. Furthermore, by guiding the filtering process to employ different filtering mechanisms in different saliency regions, the problem of blurred details and loss of coding information due to excessive filtering of the foreground target, leading to poor image clarity during video decoding, is avoided. It also prevents excessive noise in smooth background areas of the image due to insufficient filtering strength, which increases redundant coding information, reduces coding efficiency, and results in high video noise and quality degradation during video decoding.

[0075] In step S310, a visual saliency heatmap of the current frame image is obtained, and the image to be processed is filtered using the visual saliency heatmap of the current frame image to obtain the target image.

[0076] In one exemplary embodiment, obtaining a visual saliency heatmap of the current frame image may include obtaining a visual saliency heatmap of the input image of the current frame or obtaining a visual saliency heatmap of the image to be processed in the current frame. The visual saliency heatmap may have different colors, and different colors may represent different saliency scores. That is, the input image or the image to be processed in the current frame may be divided into different regions by different colors, and then the saliency score of each region is determined. The saliency score may be related to human eye attention. That is, regions with high human eye attention, mainly foreground objects with more details, have higher saliency scores, while regions with low human eye attention, mainly smooth backgrounds, have lower saliency scores.

[0077] In this example implementation, refer to Figure 4 As shown, when obtaining the visual saliency heatmap of the input image of the current frame, a first neural network model 430 with an attention mechanism trained can be obtained first. The first neural network model 430 can be a CNN network model based on Unet and the attention mechanism. The input of the first neural network model 430 is the brightness image of the input image 410 of the current frame. The brightness channel image is directly used as the input of the first neural network model 430. The output of the first neural network model 430 is the visual saliency heatmap 440 of the input image 410 of the current frame. First, the brightness image of the input image of the current frame can be extracted 420 through the Y channel. Then, the brightness image of the input image of the current frame is input to the first neural network model to obtain the visual saliency heatmap of the input image of the current frame. Then, the visual saliency heatmap of the input image of the current frame can be used to filter the input image of the current frame 450 to obtain the target image 460.

[0078] In this example implementation, the filtering mechanism for each region in the current frame input image can be determined by the saliency score of each region in the visual saliency heatmap. Regions with high saliency scores can be filtered with low intensity to preserve details as much as possible, while regions with low saliency scores can be filtered with high intensity to remove blockage artifacts. This avoids the problem of blurred details and loss of encoded information due to excessive filtering of foreground objects, resulting in poor image sharpness during video decoding. It also avoids the problem of excessive noise in smooth background areas of the image due to insufficient filtering intensity, which increases redundant encoded information, reduces coding efficiency, and leads to high video noise and quality degradation during video decoding.

[0079] In this example implementation, the saliency heatmap can be partitioned according to the colors in the saliency heatmap. For example, the colors of the visual saliency heatmap can include red, yellow, green, and blue, and the color settings can be configured when training the first neural network model. Other colors may also be included, but are not specifically limited in this real-time example.

[0080] Different salience scores can be assigned to the corresponding areas of the different colors mentioned above. For example, the salience score of the red area can be set to 100, the salience score of the yellow area can be set to 75, the score of the area between yellow and red can be greater than 75 and less than 100, the score of the green area can be set to 50, the score of the light blue area can be set to 25, and the score of the dark blue area can be set to 0.

[0081] In one example embodiment of this disclosure, the two colors can also be set to the same area. For example, red and yellow can be set to the same area, and green and blue can be set to the same area. This example embodiment does not impose any specific limitations.

[0082] When performing filtering, the unit filtering region can be determined first. The size of the unit filtering region can be customized by the user and is not specifically limited in this example implementation. The proportion of each color in the unit filtering region is determined, and the salience score corresponding to the color with the largest color proportion is determined as the salience score of the unit filtering region. Then, the filtering mechanism of each unit filtering region is determined based on the salience score.

[0083] Specifically, a first-level filtering mechanism can be used for unit filtering regions with saliency scores of 75-100, a second-level filtering mechanism for unit filtering regions with saliency scores of 50-75, a third-level filtering mechanism for unit filtering regions with saliency scores of 25-50, and a fourth-level filtering mechanism for unit filtering regions with saliency scores of 0-25. The filtering intensity of the first-level mechanism is lower than that of the second-level mechanism, which in turn is lower than that of the third-level mechanism, and so on, until the fourth-level mechanism is reached. The filters used can be those based on coding standard frameworks in related technologies, such as the H.246 framework. These filters include adjustable filtering intensity parameters, allowing different filtering mechanisms to be applied to regions with varying saliency scores. The filtering intensity refers to the fineness of removing block artifacts from an image. The higher the fineness of the removal of block artifacts, that is, the higher the accuracy of the removal of block artifacts, the higher the filtering intensity.

[0084] When acquiring the visual saliency heatmap of the image to be processed in the current frame, refer to Figure 5 As shown, steps S510 to S530 may be included.

[0085] In step S510, the input image of the current frame and the reference prediction image of the current frame are obtained.

[0086] In this example implementation, the reference prediction image of each current frame can be obtained first from the input image of the current frame. The reference prediction image of the current frame can be obtained by performing motion estimation on the image to be processed in the previous frame and the input image of the current frame, or by performing motion estimation on the input image of the current frame and the input image of the previous frame. No specific limitation is made in this example implementation.

[0087] In step S520, the decoded differential image is obtained by decoding the encoded data of the differential image between the input image of the current frame and the reference prediction image.

[0088] In this example implementation, after obtaining the reference prediction image, the input image of the current frame and the reference prediction image can be used to obtain the difference image between the input image of the current frame and the reference prediction image. During encoding, the difference image is encoded. Therefore, when encoding the next frame, the image to be processed of the previous frame is used as a reference image to obtain the target prediction image of the next frame. Therefore, the encoded data of the encoded difference image can be decoded to obtain the decoded difference image.

[0089] In this example implementation, the encoding process may include transformation and quantization. During encoding, the controller can provide encoding parameters. During transformation, the difference image can be transformed according to the size unit of the transformation block contained in the encoding parameters, such as DCT (Discrete Cosine Transform) or orthogonal transformation processing such as KL transform, which has been pre-designed based on a specific learning series. Quantization can be performed on the difference image according to the linkage parameters in the above-mentioned encoding parameters, and then the quantized transformation coefficients are used as the compression encoded data output. Quantization and transformation are commonly used techniques in related coding techniques, and will not be elaborated further here.

[0090] In this example embodiment, decoding the encoded data of the above-mentioned differential image to obtain a differential image may include performing inverse quantization and inverse transformation on the encoded data. Specifically, the encoded data is inverse quantized according to the quantization parameters of the above-mentioned encoding parameters, and inverse transformation processing (such as inverse DCT (inverse discrete cosine transform), inverse KL transform, etc.) is performed on the inverse quantized encoded data according to the size unit of the change block in the above-mentioned encoding parameters, thereby obtaining the above-mentioned decoded differential image.

[0091] In step S530, the image to be processed is obtained based on the decoded differential image and the reference prediction image.

[0092] In this example embodiment, the decoded differential image can be added to the obtained reference prediction image to obtain the image to be processed.

[0093] In this example implementation, refer to Figure 4 As shown, a pre-trained attention mechanism first neural network model 430 can be obtained first. The input of the first neural network model 430 is the brightness image of the image to be processed 410 in the current frame, and the output can be the visual saliency heatmap 440 of the image to be processed in the current frame. The brightness image of the image to be processed in the current frame can be extracted 420 through the Y channel and input into the first neural network model to obtain the visual saliency heatmap of the image to be processed in the current frame. Then, the image to be processed in the current frame can be filtered 450 using the data saliency heatmap of the current frame to obtain the target image 460.

[0094] In this example implementation, the filtering mechanism for each region in the current frame's image to be processed can be determined based on the saliency score of each region in the visual saliency heatmap. Regions with high saliency scores can be filtered with low intensity to preserve details as much as possible, while regions with low saliency scores can be filtered with high intensity to remove blockage artifacts. This avoids the problem of blurred details and loss of encoded information due to excessive filtering of foreground objects, leading to poor image clarity during video decoding. It also avoids the problem of excessive noise in smooth background areas of the image due to insufficient filtering intensity, which increases redundant encoded information, reduces coding efficiency, and results in high video noise and quality degradation during video decoding.

[0095] In this example implementation, the saliency heatmap can be partitioned according to the colors in the saliency heatmap. For example, the colors of the visual saliency heatmap can include red, yellow, green, and blue, and the color settings can be configured when training the first neural network model. Other colors may also be included, but are not specifically limited in this real-time example.

[0096] Different salience scores can be assigned to the corresponding areas of the different colors mentioned above. For example, the salience score of the red area can be set to 100, the salience score of the yellow area can be set to 75, the score of the area between yellow and red can be greater than 75 and less than 100, the score of the green area can be set to 50, the score of the light blue area can be set to 25, and the score of the dark blue area can be set to 0.

[0097] In one example embodiment of this disclosure, the two colors can also be set to the same area. For example, red and yellow can be set to the same area, and green and blue can be set to the same area. This example embodiment does not impose any specific limitations.

[0098] When performing filtering, the unit filtering region can be determined first. The size of the unit filtering region can be customized by the user and is not specifically limited in this example implementation. The proportion of each color in the unit filtering region is determined, and the salience score corresponding to the color with the largest color proportion is determined as the salience score of the unit filtering region. Then, the filtering mechanism of each unit filtering region is determined based on the salience score.

[0099] Specifically, a first-level filtering mechanism can be used for unit filtering regions with saliency scores of 75-100, a second-level filtering mechanism for unit filtering regions with saliency scores of 50-75, a third-level filtering mechanism for unit filtering regions with saliency scores of 25-50, and a fourth-level filtering mechanism for unit filtering regions with saliency scores of 0-25. The filtering intensity of the first-level mechanism is lower than that of the second-level mechanism, which in turn is lower than that of the third-level mechanism, and so on, until the fourth-level mechanism is reached. The filters used can be those based on coding standard frameworks in related technologies, such as the H.246 framework. These filters include adjustable filtering intensity parameters, allowing different filtering mechanisms to be applied to regions with varying saliency scores.

[0100] In step S320, the motion estimation vector and the target prediction image of the next frame input image are obtained using the target image and the next frame input image.

[0101] In this example embodiment, after obtaining the target image, motion estimation can be performed using the target image and the input image of the next frame to obtain the motion estimation vector and the target prediction image of the input image of the next frame.

[0102] Specifically, refer to Figure 6 As shown, a pre-trained second neural network model 650 can be obtained first. The second neural network model can be a CNN network model based on multi-scale Unet and 3D convolution. The input of the second neural network model can be the next frame input image 620 and the target image 610, as well as the visual saliency heatmap 640 of the next frame input image and the visual saliency heatmap 630 of the target image. The input is the residual vector map 660 of the next frame input image and the target image.

[0103] In this example embodiment, the target image and the next frame input image, as well as the visual saliency heatmap of the next frame input image and the visual saliency heatmap of the target image, can be input into the second neural network model to obtain the residual vector map of the next frame input image and the target image.

[0104] In this example implementation, after obtaining the residual vector map, the motion estimation vector between the next frame input image and the target image can be calculated using the residual vector map. The residual vector map includes the motion process between the target image and the next frame input image. The motion estimation vector is the relative displacement and direction of the displacement between the target image and the next frame input image, that is, the motion information between the target image and the next frame input image.

[0105] In this example implementation, the target prediction image of the next frame can be obtained based on the target image and the aforementioned motion estimation vector. Since the input to the second neural network model is a pixel-level residual vector map, that is, each pixel can correspond to a set of inter-frame motion residuals, frame prediction and motion compensation are performed based on the residual vector map output by the model, resulting in a relatively higher accuracy of the reconstructed image frame and a smoother transition of pixel motion between frames.

[0106] In another exemplary embodiment of this disclosure, the target prediction image may be obtained by obtaining the inter-frame prediction of the next frame image. The specific details of the inter-frame prediction are already quite common in related technologies, so they will not be repeated here.

[0107] It should be noted that both the first and second neural network models described above can be based on feedforward neural networks. A feedforward network can be implemented as an acyclic graph, where nodes are arranged in layers. Typically, a feedforward network topology includes an input layer and an output layer, separated by at least one hidden layer. The hidden layer transforms the input received from the input layer into a representation useful for generating the output in the output layer. Network nodes are fully connected to nodes in adjacent layers via edges, but there are no edges between nodes within a single layer. Data received at nodes in the input layer of the feedforward network is propagated (i.e., “feedforward”) to nodes in the output layer via an activation function that calculates the state of nodes in each consecutive layer of the network based on coefficients (“weights”), which are associated with each of the edges connecting these layers. The first and second neural network models can also include other neural network models, such as recurrent neural network (RNN) models, generative adversarial network (GAN) models, but are not limited thereto; other neural network models known to those skilled in the art can also be employed.

[0108] In this real-time example, the first and second neural network models described above are obtained through training. Specifically, this may include the following steps: selecting a network topology; using a set of training data representing the problem being modeled by the network; and adjusting the weights until the network model exhibits minimum error for all instances in the training dataset. For example, during supervised learning training for a neural network, the output generated by the network in response to inputs representing instances in the training dataset is compared to the “correct” labeled output of that instance; an error signal representing the difference between the output and the labeled output is calculated; and the weights associated with the connections are adjusted to minimize the error as the error signal is backpropagated through the layers of the network.

[0109] In step S330, the difference image and motion estimation vector between the next frame input image and the target prediction image are encoded.

[0110] In this example implementation, after obtaining the target prediction image, a difference image can be obtained by subtracting the target prediction image from the next frame input image. The obtained difference image and the motion estimation vector are then encoded, wherein the encoding includes variation and quantization. The specific details of variation and brightness have been described in detail above, and therefore will not be repeated here.

[0111] This disclosure also provides a decoding method, referring to... Figure 7 As shown, the decoding method may include the following steps:

[0112] Step S710: Obtain the input image of the current frame and the reference prediction image of the current frame;

[0113] Step S720: Decode the encoded data of the difference image between the input image and the reference prediction image of the current frame to obtain the decoded difference image;

[0114] Step S730: Obtain the image to be processed based on the decoded differential image and the reference prediction image;

[0115] Step S740: Obtain the saliency heatmap of the image to be processed in the current frame, and use the visual saliency heatmap of the image to be processed in the current frame to perform filtering processing on the image to be processed to obtain the output image of the current frame.

[0116] The steps described above are explained in detail below.

[0117] In step S710, the reference prediction image of the current frame is obtained.

[0118] In one example embodiment of this disclosure, the processor may first obtain a reference prediction image of the current frame. Obtaining the reference prediction image of the current frame may be achieved by using the output image of the previous frame and the motion estimation vector of the previous frame.

[0119] In this example implementation, since the previous frame image has already been decoded, the output image of the previous frame image and the motion estimation vector between the previous frame image and the current frame image can be obtained. Then, the reference prediction image of the current frame can be obtained by using motion compensation based on the motion estimation vector and the output image of the previous frame.

[0120] In step S720, the decoded differential image is obtained by decoding the encoded data of the differential image between the input image of the current frame and the reference prediction image.

[0121] In one example embodiment of this disclosure, after obtaining the reference prediction image of the current frame, the encoded data of the difference image between the input image and the reference prediction image are decoded to obtain a decoded difference image, wherein the difference image is the input image to be processed. After obtaining the decoded difference image, step S730 can be executed. The decoding process includes inverse transformation and inverse quantization. The specific details of inverse transformation and inverse quantization have been described in detail in the above image encoding method, so they will not be repeated here.

[0122] In step S730, the image to be processed is obtained based on the decoded differential image and the reference prediction image;

[0123] In step S740, a saliency heatmap of the image to be processed in the current frame is obtained, and the image to be processed is filtered using the visual saliency heatmap of the image to be processed in the current frame to obtain the output image of the current frame.

[0124] In this example embodiment, the image to be processed can be obtained by adding the decoded differential image and the reference prediction image. Then, the saliency heatmap of the image to be processed can be obtained, and the image to be processed can be filtered using the visual saliency heatmap of the image to be processed in the current frame to obtain the output image of the current frame.

[0125] Specifically, in this example implementation, refer to Figure 4As shown, the input of the first neural network model 430 is the brightness value of the image 410 to be processed in the current frame, and the output can be the visual saliency heatmap 440 of the image 410 to be processed in the current frame. First, the brightness image of the image to be processed in the current frame can be extracted using the Y channel 420, that is, the image of the Y channel is used as the input of the first neural network model to obtain the visual saliency heatmap of the image to be processed in the current frame. Then, the image to be processed in the current frame can be filtered 450 using the data saliency heatmap of the current frame to obtain the output image 460 of the current frame.

[0126] In this example implementation, the filtering mechanism for each region in the current frame's image to be processed can be determined by the saliency score of each region in the visual saliency heatmap. Regions with high saliency scores can be filtered with low intensity to preserve details as much as possible, while regions with low saliency scores can be filtered with high intensity to remove blockage artifacts as much as possible. This avoids the problem of blurred details and reduced image sharpness during video decoding caused by excessive filtering of foreground objects; it also avoids the problem of high video noise and quality degradation during video decoding.

[0127] In summary, this exemplary embodiment utilizes the visual saliency heatmap of the current frame to filter the image, thereby improving the display effect of the target image, enhancing the accuracy of the target prediction image, and improving coding accuracy. Furthermore, by employing different filtering mechanisms in different saliency regions during the guided filtering process, the problem of blurred details and loss of coding information due to excessive filtering of the foreground target, leading to poor image clarity during video decoding, is avoided. It also prevents excessive noise in smooth background areas of the image due to insufficient filtering intensity, which increases redundant coding information, reduces coding efficiency, and results in high video noise and quality degradation during video decoding.

[0128] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0129] Further reference Figure 8 As shown, this example embodiment also provides an image encoding device 800, including a filtering module 810, a prediction module 820, and an encoding module 830.

[0130] The filtering module 810 can be used to obtain the visual saliency heatmap of the current frame image and use the visual saliency heatmap of the current frame image to filter the previous frame image to obtain the target image; the prediction module 820 can be used to obtain the motion estimation vector and the target prediction image of the next frame input image using the target image and the next frame input image; wherein the above-mentioned encoding module 830 can be used to encode the difference image and motion estimation vector of the next frame input image and the target prediction image.

[0131] In this example embodiment, the filtering module may include a heatmap acquisition module and a deblocking filtering module. The heatmap acquisition module is used to acquire the visual saliency heatmap of the current frame image; the deblocking filtering module is used to filter the previous frame image using the visual saliency heatmap of the current frame image to obtain the target image.

[0132] In this example embodiment, the heatmap acquisition module may include a first neural network model with a pre-trained attention mechanism to acquire a visual saliency heatmap of the image in the current frame. The method of acquiring a data saliency heatmap using the first neural network model has been described in detail in the previous article, so it will not be repeated here.

[0133] In this example embodiment, the prediction module may include a motion estimation module and a motion compensation module, wherein the motion estimation module is used to obtain a motion estimation vector using the target image and the next frame input image; the motion compensation module is used to obtain a target prediction image of the next frame input image based on the target image and the motion estimation vector.

[0134] In this example implementation, the encoding module may include a difference module and an image encoding module. The difference module is used to obtain a difference image between the next frame input image and the target prediction image; the image encoding module is used to encode the difference image between the next frame input image and the target prediction image and the motion estimation vector.

[0135] In one exemplary embodiment of this disclosure, reference is made to Figure 9As shown, the coding control unit 901 performs the following processing: determining the maximum size of the coding block that serves as the processing unit when performing motion compensation prediction processing (inter-frame prediction processing) or intra-frame prediction processing (intra-frame prediction processing), and determining the upper limit number of levels when the coding block with the maximum size is segmented hierarchically. Furthermore, the coding control unit 901 performs the following processing: selecting a coding mode suitable for hierarchically segmented coding blocks from one or more available coding modes (one or more intra-frame coding modes, one or more inter-frame coding modes). The block segmentation unit 902 performs the following processing: if an image signal representing an input image is input, it segments the input image represented by the image signal into coding blocks of the maximum size determined by the coding control unit 901, and segments the coding block hierarchically until the upper limit number of levels determined by the coding control unit 901 is reached.

[0136] The first switching switch 903 performs the following processing: if the coding mode selected by the coding control unit 901 is the intra-frame coding mode, the coding block divided by the block segmentation unit 902 is output to the intra-frame prediction unit 904; if the coding mode selected by the coding control unit 901 is the inter-frame coding mode, the coding block divided by the block segmentation unit 902 is output to the motion compensation prediction unit 905.

[0137] The intra-frame prediction unit 904 performs the following processing: If a coded block segmented by the block segmentation unit 902 is received from the first switching switch 903, and the coded block becomes a coded block after passing through the transform / quantization unit 907, the intra-frame information acquisition unit 910 acquires the image to be processed of the coded block. That is, the input of the intra-frame information acquisition unit 910 is the image obtained by the coded block through the inverse quantization / inverse transform unit 908 and the addition unit. The intra-frame prediction unit 904 then performs intra-frame prediction processing for the coded block by referring to the image to be processed of the coded block (referencing the image) and using the intra-frame prediction parameters output from the coding control unit 901, and generates a prediction image.

[0138] The motion compensation prediction unit 905 performs the following processing: If a coded block segmented by the block segmentation unit 902 is received from the first switching switch 903, the motion estimation unit 912 calculates a motion estimation vector by comparing the local images to be processed of the coded block and the coded block, and the motion compensation prediction unit 905 uses the motion estimation vector and the inter-frame prediction parameters output from the coding control unit 901 to perform inter-frame prediction processing (motion compensation prediction processing) for the coded block to generate a prediction image. The coded block becomes a coded block after passing through the transform / quantization unit 907, and the coded block obtains the image to be processed through the inverse quantization / inverse transform unit 908 and the addition unit.

[0139] The motion estimation unit 912 can be a motion estimation module, and the motion compensation prediction unit 905 can be a motion compensation module. The prediction module includes the motion compensation prediction unit 905 and the intra-frame prediction unit 904.

[0140] The subtraction unit 906 performs the following processing: by subtracting the prediction image generated by the intra-frame prediction unit 904 or the motion compensation prediction unit 905 from the coded block segmented by the block segmentation unit 902, a difference image (= coded block - prediction image) is generated.

[0141] The transform / quantization unit 907 performs the following processing: According to the transform block size unit included in the encoding parameters output from the encoding control unit 901, it performs transform processing on the difference image generated by the subtraction unit 906 (e.g., DCT (Discrete Cosine Transform), or orthogonal transform processing such as KL transform pre-designed for a specific learning series), and uses the quantization parameters included in the encoding parameters to quantize the transform coefficients of the difference image, thereby outputting the quantized transform coefficients as compressed data of the difference image. The encoding module described above includes the subtraction unit 906 and the transform / quantization unit 907.

[0142] The inverse quantization / inverse transform unit 908 performs the following processing: using the quantization parameters included in the encoding parameters output from the encoding control unit 901, it performs inverse quantization on the compressed data output from the transform / quantization unit 907, and performs inverse transform processing (e.g., inverse DCT (inverse discrete cosine transform), inverse KL transform, etc.) on the inverse quantized compressed data according to the transform block size unit included in the encoding parameters, thereby outputting the compressed data after inverse transform processing as a local decoding prediction differential signal (representing the data of the decompressed differential image).

[0143] The addition unit 909 performs the following processing: by adding the image to be processed output from the inverse quantization / inverse transform unit and the prediction image generated by the intra-frame prediction unit 904 or the motion compensation prediction unit 905, a target image is generated.

[0144] The loop filter unit 911 performs the following processing: by performing filtering processing (loop filtering processing) on ​​the image to be processed generated by the addition unit 909, distortion (block distortion) occurring in block boundaries is removed.

[0145] When removing block distortion in a local area of ​​the image to be processed, the loop filter 911 sets the intensity of the block distortion removal filter according to the signal components (luminance signal component, chrominance signal component) in accordance with the encoding mode (intra-frame encoding mode, inter-frame encoding mode) selected by the encoding control unit 901. The aforementioned filtering module includes a loop filter.

[0146] The above-mentioned encoding module may also include an entropy encoding unit 913 that performs the following processing: performing variable-length encoding on the compressed data output from the transform / quantization unit 907, the encoding mode and encoding parameters output from the encoding control unit 901, and the intra-frame prediction parameters output from the intra-frame prediction unit 904 or the inter-frame prediction parameters (including motion estimation vectors) output from the motion compensation prediction unit 905, to generate a bitstream of encoded data that reuses the compressed data, encoding mode, prediction differential encoding parameters, intra-frame prediction parameters / inter-frame prediction parameters.

[0147] An example is envisioned where the encoding control unit 901, block segmentation unit 902, first switching switch 903, intra-frame prediction unit 904, motion compensation prediction unit 905, subtraction unit 906, transform / quantization unit 907, inverse quantization / inverse transform unit 908, addition unit 909, intra-frame information acquisition unit 910, loop filtering unit 911, motion estimation unit 912, and entropy coding unit 913, which are components of an image encoding device, are each composed of dedicated hardware (e.g., a semiconductor integrated circuit with a CPU installed, or a single-chip microcomputer, etc.). However, in the image... When the encoding device is composed of a computer, a program describing the processing contents of the encoding control unit 901, block segmentation unit 902, first switching switch 903, intra-frame prediction unit 904, motion compensation prediction unit 905, subtraction unit 906, transform / quantization unit 907, inverse quantization / inverse transform unit 908, addition unit 909, intra-frame information acquisition unit 910, loop filtering unit 911, motion estimation unit 912, and entropy coding unit 913 can be stored in the computer's memory, and the computer's CPU executes the program stored in the memory.

[0148] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0149] Reference Figure 10 As shown, this disclosure also provides an image decoding apparatus 1000, including an acquisition module 1010, a decoding module 1020, a generation module 1030, and a filtering module 1040. The acquisition module 1010 is used to acquire a reference prediction image of the current frame; the decoding module 1020 is used to decode the encoded data of the difference image between the input image and the reference prediction image of the current frame to obtain a decoded difference image; the generation module 1030 is used to obtain an image to be processed based on the decoded difference image and the reference prediction image; and the filtering module 1040 is used to acquire a saliency heatmap of the image to be processed in the current frame, and to perform filtering processing on the image to be processed using the visual saliency heatmap of the image to be processed in the current frame to obtain an output image of the current frame.

[0150] It should be noted that the filtering module in the image decoding device and the image encoding device described above can be the same module or different modules, and no specific limitation is made in this example embodiment.

[0151] In this example embodiment, the filtering module includes a heatmap acquisition module and a deblocking filtering module. The heatmap acquisition module is used to acquire the saliency heatmap of the image to be processed in the current frame; the deblocking filtering module is used to filter the image to be processed using the visual saliency heatmap of the image to be processed in the current frame to obtain the output image of the current frame.

[0152] Specifically, the heatmap acquisition module may include a pre-trained first neural network model 430 with an attention mechanism, as shown in reference to... Figure 4 As shown, the input of the first neural network model 430 is the brightness value of the image 410 to be processed in the current frame, and the output can be the visual saliency heatmap 440 of the image 410 to be processed in the current frame. First, the brightness value of the image 410 to be processed in the current frame can be extracted 420 through the Y channel, and the brightness value can be input into the first neural network model to obtain the visual saliency heatmap of the image 410 to be processed in the current frame. Then, the image 410 to be processed in the current frame can be filtered 450 using the data saliency heatmap of the current frame to obtain the output image 460 of the current frame.

[0153] In this example implementation, the filtering mechanism for each region in the current frame's image to be processed can be determined by the saliency score of each region in the visual saliency heatmap. Regions with high saliency scores can be filtered with low intensity to preserve details as much as possible, while regions with low saliency scores can be filtered with high intensity to remove blockage artifacts as much as possible. This avoids the problem of blurred details and reduced image sharpness during video decoding caused by excessive filtering of foreground objects; it also avoids the problem of high video noise and quality degradation during video decoding.

[0154] In one exemplary embodiment of this disclosure, reference is made to Figure 11 As shown, the coding control unit 901 performs the following processing: determining the maximum size of the coding block that becomes the processing unit when performing motion compensation prediction processing (inter-frame prediction processing) or intra-frame prediction processing (intra-frame prediction processing), and determining the upper limit of the number of levels when the maximum-sized coding block is divided into layers. Furthermore, the coding control unit 901 performs the following processing: selecting a coding mode suitable for layered division of each coding block from one or more available coding modes (one or more intra-frame coding modes and one or more inter-frame coding modes).

[0155] The first switching switch 903 performs the following processing: if the coding mode selected by the coding control unit 901 is the intra-frame coding mode, the coding block divided by the block segmentation unit 902 is output to the intra-frame prediction unit 904; if the coding mode selected by the coding control unit 901 is the inter-frame coding mode, the coding block divided by the block segmentation unit 902 is output to the motion compensation prediction unit 905.

[0156] The intra-frame prediction unit 904 performs the following processing: If a coded block segmented by the block segmentation unit 902 is received from the first switching switch 903, and the coded block becomes a coded block after passing through the transform / quantization unit 907, the intra-frame information acquisition unit 910 acquires the image to be processed of the coded block. That is, the input of the intra-frame information acquisition unit 910 is the image obtained by the coded block through the inverse quantization / inverse transform unit 908 and the addition unit. The intra-frame prediction unit 904 then performs intra-frame prediction processing for the coded block by referring to the image to be processed of the coded block (referencing the image) and using the intra-frame prediction parameters output from the coding control unit 901, and generates a prediction image.

[0157] The motion compensation prediction unit 905 performs the following processing: If a coded block segmented by the block segmentation unit 902 is received from the first switching switch 903, the motion estimation unit 912 calculates a motion estimation vector by comparing the local images to be processed of the coded block and the coded block, and the motion compensation prediction unit 905 uses the motion estimation vector and the inter-frame prediction parameters output from the coding control unit 901 to perform inter-frame prediction processing (motion compensation prediction processing) for the coded block to generate a prediction image. The coded block becomes a coded block after passing through the transform / quantization unit 907, and the coded block obtains the image to be processed through the inverse quantization / inverse transform unit 908 and the addition unit.

[0158] In the decoding device, the acquisition module includes the motion compensation prediction unit 905 and the intra-frame prediction unit 904.

[0159] The inverse quantization / inverse transform unit performs the following processing: using the quantization parameters included in the encoding parameters output from the encoding control unit 901, the input compressed data is inversely quantized, and inverse transform processing (such as inverse DCT (inverse discrete cosine transform), inverse KL transform, etc.) is performed on the inverse quantized compressed data according to the transform block size unit included in the encoding parameters, thereby outputting the compressed data after inverse transform processing as a local decoding prediction differential signal (representing the data of the decompressed differential image).

[0160] The decoding module in the image decoding device includes the aforementioned inverse quantization / inverse transform unit 908.

[0161] The addition unit 909 performs the following processing: by adding the image to be processed, which is output from the inverse quantization / inverse transform unit, to the predicted image generated by the intra-frame prediction unit 904 or the motion compensation prediction unit 905, a target image is generated. The generation module in the aforementioned image decoding apparatus includes the addition unit 909.

[0162] The loop filter unit 911 performs the following processing: by performing filtering processing (loop filtering processing) on ​​the image to be processed generated by the addition unit 909, distortion (block distortion) occurring in block boundaries is removed.

[0163] When removing block distortion in a local image to be processed, the loop filter 911 sets the intensity of the filtering to remove block distortion according to the signal components (luminance signal component, chrominance signal component) in accordance with the encoding mode (intra-frame encoding mode, inter-frame encoding mode) selected by the encoding control unit 901.

[0164] The filtering module in the above-mentioned decoding device includes a loop filter unit 911.

[0165] An example is envisioned where the encoding control unit 901, first switching switch 903, intra-frame prediction unit 904, motion compensation prediction unit 905, inverse quantization / inverse transform unit 908, addition unit 909, intra-frame information acquisition unit 910, and loop filtering unit 911, which are components of an image encoding apparatus, are each composed of dedicated hardware (e.g., a semiconductor integrated circuit with a CPU installed, or a single-chip microcomputer). However, if the image encoding apparatus is composed of a computer, a program describing the processing content of the encoding control unit 901, block segmentation unit 902, first switching switch 903, intra-frame prediction unit 904, motion compensation prediction unit 905, inverse quantization / inverse transform unit 908, addition unit 909, intra-frame information acquisition unit 910, and loop filtering unit 911 can be stored in the memory of the computer, and the CPU of the computer executes the program stored in the memory.

[0166] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0167] Reference Figure 12As shown, this disclosure also provides a codec 1200, including a decoding module 1210, a generation module 1220, a filtering module 1230, a prediction module 1240, and an encoding module 1250. The decoding module 1210 is used to decode the encoded data of the difference image between the input image of the current frame and the reference prediction image of the current frame to obtain a decoded difference image; the generation module 1220 is used to obtain a to-be-processed image based on the decoded difference image and the reference prediction image; the filtering module 1230 is used to obtain a visual saliency heatmap of the to-be-processed image of the current frame, and use the visual saliency heatmap of the to-be-processed image of the current frame to filter the to-be-processed image to obtain a target image; the prediction module 1240 is used to obtain a motion estimation vector and a target prediction image of the next frame input image using the target image and the next frame input image; the encoding module 1250 is used to encode the difference image between the next frame input image and the target prediction image and the motion estimation vector.

[0168] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0169] In one exemplary embodiment of this disclosure, reference is made to Figure 13 As shown, the coding control unit 901 performs the following processing: determining the maximum size of the coding block that serves as the processing unit when performing motion compensation prediction processing (inter-frame prediction processing) or intra-frame prediction processing (intra-frame prediction processing), and determining the upper limit number of levels when the maximum-sized coding block is segmented hierarchically. Furthermore, the coding control unit 901 performs the following processing: selecting a coding mode suitable for hierarchically segmented coding blocks from one or more available coding modes (one or more intra-frame coding modes, one or more inter-frame coding modes). The block segmentation unit 902 performs the following processing: if an image signal representing an input image is input, it segments the input image represented by the image signal into coding blocks of the maximum size determined by the coding control unit 901, and segments the coding block hierarchically until the upper limit number of levels determined by the coding control unit 901 is reached.

[0170] The second switching switch 914 is used to determine whether the codec is in encoding or decoding mode. If the second switching switch 914 directs data to the block segmentation unit 902, the codec is in encoding mode; if the second switching switch 914 directly directs data to the inverse quantization unit, the codec is in decoding mode.

[0171] The first switching switch 903 performs the following processing: if the coding mode selected by the coding control unit 901 is the intra-frame coding mode, the coding block divided by the block segmentation unit 902 is output to the intra-frame prediction unit 904; if the coding mode selected by the coding control unit 901 is the inter-frame coding mode, the coding block divided by the block segmentation unit 902 is output to the motion compensation prediction unit 905.

[0172] The intra-frame prediction unit 904 performs the following processing: If a coded block segmented by the block segmentation unit 902 is received from the first switching switch 903, and the coded block becomes a coded block after passing through the transform / quantization unit 907, the intra-frame information acquisition unit 910 acquires the image to be processed of the coded block. That is, the input of the intra-frame information acquisition unit 910 is the image obtained by the coded block through the inverse quantization / inverse transform unit 908 and the addition unit. The intra-frame prediction unit 904 then performs intra-frame prediction processing for the coded block by referring to the image to be processed of the coded block (referencing the image) and using the intra-frame prediction parameters output from the coding control unit 901, and generates a prediction image.

[0173] The motion compensation prediction unit 905 performs the following processing: If a coded block segmented by the block segmentation unit 902 is received from the first switching switch 903, the motion estimation unit 912 calculates a motion estimation vector by comparing the local images to be processed of the coded block and the coded block, and then uses the motion estimation vector and the inter-frame prediction parameters output from the coding control unit 901 to perform inter-frame prediction processing (motion compensation prediction processing) for the coded block to generate a prediction image. The coded block becomes a coded block after passing through the transform / quantization unit 907, and the coded block obtains the image to be processed through the inverse quantization / inverse transform unit 908 and the addition unit. The prediction module includes the motion compensation prediction unit 905 and the intra-frame prediction unit 904.

[0174] The subtraction unit 906 performs the following processing: by subtracting the prediction image generated by the intra-frame prediction unit 904 or the motion compensation prediction unit 905 from the coded block segmented by the block segmentation unit 902, a difference image (coded block - prediction image) is generated.

[0175] The transform / quantization unit 907 performs the following processing: according to the transform block size unit included in the encoding parameters output from the encoding control unit 901, it performs transform processing on the difference image generated by the subtraction unit 906 (e.g., DCT (Discrete Cosine Transform), KL transform, or other orthogonal transform processing with a pre-designed basic series of specific learning), and uses the quantization parameters included in the encoding parameters to quantize the transform coefficients of the difference image, thereby outputting the quantized transform coefficients as compressed data of the difference image.

[0176] The above-mentioned encoding module is composed of the subtraction unit 906 and the transformation / quantization unit 907.

[0177] The inverse quantization / inverse transform unit 908 performs the following processing: using the quantization parameters included in the encoding parameters output from the encoding control unit 901, it performs inverse quantization on the compressed data output from the transform / quantization unit 907, and performs inverse transform processing (e.g., inverse DCT (inverse discrete cosine transform), inverse KL transform, etc.) on the inverse quantized compressed data according to the transform block size unit included in the encoding parameters, thereby outputting the compressed data after inverse transform processing as a local decoding prediction differential signal (representing the data of the decompressed differential image).

[0178] The aforementioned decoding module includes an inverse quantization / inverse transform unit 908.

[0179] The addition unit 909 performs the following processing: by adding the image to be processed, which is output from the inverse quantization / inverse transform unit, to the predicted image generated by the intra-frame prediction unit 904 or the motion compensation prediction unit 905, a target image is generated. The above-mentioned generation module includes the addition unit 909.

[0180] The loop filter unit 911 performs the following processing: by performing filtering processing (loop filtering processing) on ​​the image to be processed generated by the addition unit 909, distortion (block distortion) occurring in block boundaries is removed.

[0181] When removing block distortion in a local image to be processed, the loop filter 911 sets the intensity of the filtering to remove block distortion according to the signal components (luminance signal component, chrominance signal component) in accordance with the encoding mode (intra-frame encoding mode, inter-frame encoding mode) selected by the encoding control unit 901.

[0182] The aforementioned filtering module includes the aforementioned loop filtering unit 911.

[0183] The entropy coding unit performs the following processing: it performs variable-length coding on the compressed data output from the transform / quantization unit 907, the coding mode and coding parameters output from the coding control unit 901, and the intra-frame prediction parameters output from the intra-frame prediction unit 904 or the inter-frame prediction parameters (including motion estimation vectors) output from the motion compensation prediction unit 905, and generates a bitstream of coded data that reuses the compressed data, coding mode, prediction differential coding parameters, intra-frame prediction parameters / inter-frame prediction parameters.

[0184] An example is envisioned in which the components of an image encoding apparatus, namely the encoding control unit 901, block segmentation unit 902, first switching switch 903, intra-frame prediction unit 904, motion compensation prediction unit 905, subtraction unit 906, transform / quantization unit 907, inverse quantization / inverse transform unit 908, addition unit 909, intra-frame information acquisition unit 910, loop filtering unit 911, motion estimation unit 912, and entropy encoding unit 913, are each composed of dedicated hardware (e.g., a semiconductor integrated circuit with a CPU installed, or a single-chip microcomputer). However, when the image encoding device is composed of a computer, a program describing the processing contents of the encoding control unit 901, block segmentation unit 902, first switching switch 903, intra-frame prediction unit 904, motion compensation prediction unit 905, subtraction unit 906, transform / quantization unit 907, inverse quantization / inverse transform unit 908, addition unit 909, loop filtering unit 911, motion estimation unit 912, and entropy encoding unit 913 can be saved in the computer's memory, and the computer's CPU executes the program saved in the memory.

[0185] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0186] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0187] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0188] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0189] Furthermore, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0190] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0191] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image coding method, wherein, The method comprises the following steps: obtaining a visual saliency heat map of an image of a current frame, and filtering the image of the current frame by using the visual saliency heat map of the image of the current frame to obtain a target image; obtaining a motion estimation vector and a target prediction image of a next frame input image by using the target image and the next frame input image; encoding a difference image of the next frame input image and the target prediction image and the motion estimation vector; the filtering process of the image of the current frame by using the visual saliency heat map of the image of the current frame to obtain a target image comprises the following steps: determining the saliency score of each region in the visual saliency heat map of the image of the current frame; determining the filtering mechanism of each region of the image of the current frame according to the saliency score; filtering the image of the current frame according to the filtering mechanism of each region to obtain a target image; in the two regions with different saliency scores, the filtering strength of the filtering mechanism of the region with the higher saliency score is lower than that of the region with the lower saliency score.

2. The method of claim 1, wherein, The image of the current frame is the input image of the current frame.

3. The method of claim 1, wherein, The image of the current frame is the to-be-processed image of the input image of the current frame.

4. The method of claim 3, wherein, The method for obtaining the to-be-processed image of the current frame comprises the following steps: obtaining the input image of the current frame and the reference prediction image of the current frame; decoding the encoding data of the difference image of the input image of the current frame and the reference prediction image to obtain a decoded difference image; obtaining the to-be-processed image according to the decoded difference image and the reference prediction image.

5. The method of claim 4, wherein, The method for obtaining the reference prediction image of the current frame comprises the following steps: obtaining the reference prediction image of the current frame by using the motion estimation of the to-be-processed image of the previous frame and the input image of the current frame.

6. The method of claim 1, wherein, The method for obtaining the visual saliency heat map of the image of the current frame comprises the following steps: obtaining a first neural network model of a pre-trained attention mechanism; extracting the brightness information of the image of the current frame; inputting the brightness information into the first neural network model of the attention mechanism to obtain the visual saliency heat map.

7. The method of claim 1, wherein, The method for obtaining the motion estimation vector and the target prediction image of the next frame input image by using the target image and the next frame input image comprises the following steps: obtaining a second neural network model; obtaining the visual saliency heat map of the next frame input image and the visual saliency heat map of the target image; inputting the next frame input image and the target image and the visual saliency heat map of the next frame input image and the visual saliency heat map of the target image into the second neural network model to obtain a residual vector map of the next frame input image and the target image; calculating the motion estimation vector according to the residual vector map; obtaining the target prediction image of the next frame input image by using the target image and the motion estimation vector.

8. An image decoding method, wherein The method comprises the following steps: obtaining the reference prediction image of the current frame; decoding the encoding data of the difference image of the input image of the current frame and the reference prediction image to obtain a decoded difference image; obtaining the to-be-processed image according to the decoded difference image and the reference prediction image; obtaining a visual saliency heat map of the to-be-processed image of the current frame, and filtering the to-be-processed image by using the visual saliency heat map of the to-be-processed image of the current frame to obtain an output image of the current frame; Filtering the image to be processed of the current frame by using the visual saliency heat map of the image to be processed of the current frame to obtain an output image of the current frame; comprising: Determining the filtering mechanism of each region in the image to be processed of the current frame by using the saliency score of each region in the visual saliency heat map; In the two regions with different saliency scores, the filtering strength of the filtering mechanism of the region with higher saliency score is lower than that of the region with lower saliency score.

9. The method of claim 8, wherein, Obtaining the visual saliency heat map of the image to be processed of the current frame, comprising: Obtaining the first neural network model of the pre-trained attention mechanism; Extracting the brightness information of the image to be processed of the current frame; Inputting the brightness information into the first neural network model of the attention mechanism to obtain the visual saliency heat map.

10. The method of claim 8, wherein, Obtaining the reference prediction image of the current frame, comprising: Obtaining the reference prediction image by using the output image of the previous frame and the motion estimation vector of the output image of the previous frame and the input image of the current frame.

11. An encoding apparatus, comprising: Comprising: A filtering module, configured to obtain a visual saliency heat map of an image of a current frame, and filter the image of the current frame by using the visual saliency heat map of the image of the current frame to obtain a target image; A prediction module, configured to obtain a motion estimation vector and a target prediction image of a next frame input image by using the target image and the next frame input image; An encoding module, configured to encode a difference image of the next frame input image and the target prediction image and the motion estimation vector; Filtering the image of the current frame by using the visual saliency heat map of the image of the current frame to obtain a target image, comprising: Determining the saliency score of each region in the visual saliency heat map of the image of the current frame; Determining the filtering mechanism of each region in the image of the current frame according to the saliency score; Filtering the image of the current frame according to the filtering mechanism of each region to obtain a target image; In the two regions with different saliency scores, the filtering strength of the filtering mechanism of the region with higher saliency score is lower than that of the region with lower saliency score.

12. The encoding apparatus of claim 11, wherein, The filtering module comprises: A heat map obtaining module, configured to obtain a visual saliency heat map of an image of a current frame; A deblocking filtering module, configured to filter the image of the current frame by using the visual saliency heat map of the image of the current frame to obtain a target image.

13. The encoding apparatus of claim 12, wherein, The heat map obtaining module comprises a first neural network model of a pre-trained attention mechanism, and the obtaining of the visual saliency heat map of the image of the current frame comprises: Extracting the brightness information of the image of the current frame; Inputting the brightness information into the first neural network model of the attention mechanism to obtain the visual saliency heat map.

14. The encoding device of claim 11, wherein, The prediction module comprises a pre-trained second neural network model, and the obtaining of the target prediction image of the next frame input image according to the target image and the motion estimation vector comprises: Obtaining the visual saliency heat map of the next frame input image and the visual saliency heat map of the target image; inputting the next frame of input image and the target image and the visual saliency heat map of the next frame of input image and the visual saliency heat map of the target image into the second neural network model to obtain a residual vector map of the next frame of input image and the target image; calculating a motion estimation vector according to the residual vector map.

15. A decoding apparatus, wherein, comprising: an acquisition module, configured to acquire a reference prediction image of a current frame; a decoding module, configured to decode encoded data of a difference image between an input image of the current frame and the reference prediction image to obtain a decoded difference image; a generation module, configured to obtain a to-be-processed image according to the decoded difference image and the reference prediction image; a filtering module, configured to acquire a saliency heat map of the to-be-processed image of the current frame, and perform filtering processing on the to-be-processed image by using the visual saliency heat map of the to-be-processed image of the current frame to obtain an output image of the current frame; performing filtering processing on the to-be-processed image by using the visual saliency heat map of the to-be-processed image of the current frame to obtain an output image of the current frame; comprising: determining a filtering mechanism of each region in the to-be-processed image of the current frame according to a saliency value of each region in the visual saliency heat map; in two regions with different saliency values, a filtering strength of the filtering mechanism of the region with a higher saliency value is lower than a filtering strength of the filtering mechanism of the region with a lower saliency value.

16. A codec, wherein, comprising: a decoding module, configured to decode encoded data of a difference image between an input image of a current frame and a reference prediction image of the current frame to obtain a decoded difference image; a generation module, configured to obtain a to-be-processed image according to the decoded difference image and the reference prediction image; a filtering module, configured to acquire a visual saliency heat map of the to-be-processed image of the current frame, and perform filtering processing on the to-be-processed image by using the visual saliency heat map of the to-be-processed image of the current frame to obtain a target image; a prediction module, configured to acquire a motion estimation vector and a target prediction image of a next frame of input image by using the target image and the next frame of input image; an encoding module, configured to encode a difference image between the next frame of input image and the target prediction image and the motion estimation vector; performing filtering processing on the to-be-processed image by using the visual saliency heat map of the to-be-processed image of the current frame to obtain a target image, comprising: determining a saliency value of each region in the visual saliency heat map of the image of the current frame; determining a filtering mechanism of each region in the image of the current frame according to the saliency value; performing filtering processing on the image of the current frame according to the filtering mechanism of each region to obtain a target image; in two regions with different saliency values, a filtering strength of the filtering mechanism of the region with a higher saliency value is lower than a filtering strength of the filtering mechanism of the region with a lower saliency value.

17. The codec of claim 16, wherein the filtering module is further configured to acquire a saliency heat map of the to-be-processed image of the current frame, and perform filtering processing on the to-be-processed image by using the visual saliency heat map of the to-be-processed image of the current frame to obtain an output image of the current frame.

18. A computer readable storage medium having stored thereon a computer program, wherein, The program, when executed by the processor, implements the image encoding method of any one of claims 1 to 7 or the image decoding method of any one of claims 8 to 10.

19. An electronic device, comprising: Comprising: The encoding apparatus of any one of claims 11 to 14 or the decoding apparatus of claim 15 or the codec of claim 16 or 17.

Images