Processing method, processing device and storage medium

Abstract

Provided in the present application are a processing method, a processing device and a storage medium. The processing method can be applied to a processing device, and comprises: on the basis of a first parameter, determining or generating at least one predicted pixel. The technical solution of the present application can improve the accuracy of pixel prediction, and thus can improve the coding and decoding quality during video coding and / or decoding.

Description

Processing method, processing device and storage medium Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a processing method, a processing device, and a storage medium. Background Art

[0002] The existing video coding standard (H.266 / VVC) proposes a video frame encoding technology. For example, when encoding and decoding a video frame, the protocol divides each frame into different blocks and performs prediction processing and encoding and decoding processing.

[0003] During the process of conceiving and implementing this application, the inventors discovered that there are at least the following problems: in the intra-frame prediction and / or inter-frame prediction process, the prediction effect of pixels is not ideal, which leads to poor encoding and decoding quality in the video encoding and / or decoding process.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] In response to the above technical problems, the present application provides a processing method, a processing device and a storage medium, which can improve the accuracy of pixel prediction and thereby improve the encoding and decoding quality during video encoding and / or decoding.

[0006] The present application provides a processing method that can be applied to a processing device, comprising the steps of:

[0007] S10 , determining or generating at least one predicted pixel according to a first parameter.

[0008] Optionally, the first parameter includes at least one reference information of the current block and a first model.

[0009] Optionally, the at least one reference information of the current block includes pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block of the current block.

[0010] Optionally, the first model includes at least one of the following:

[0011] Neural network models;

[0012] Regression models;

[0013] Matrix-based mathematical models;

[0014] Mathematical models based on lookup tables;

[0015] Mathematical model based on normalization operation.

[0016] Optionally, the method for determining or generating at least one reference area includes at least one of the following:

[0017] Determine or generate at least one reference area according to at least one of an upper adjacent pixel, an upper non-adjacent pixel, a left adjacent pixel, a left non-adjacent pixel, an upper left adjacent pixel, and an upper left non-adjacent pixel of the current block;

[0018] Determine or generate at least one reference area according to at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block, and a default block corresponding to the current block;

[0019] Determine or generate at least one reference area according to at least one of the width, height, block size and block area of ​​the current block;

[0020] Determine or generate at least one reference area based on a candidate block determined or generated based on a candidate motion vector or a candidate block vector of a current block;

[0021] If the first information of the current block satisfies the first condition, the reference area is the first reference area;

[0022] If the first information of the current block does not meet the first condition, the reference area is the second reference area.

[0023] Optionally, step S10 includes at least one of the following:

[0024] Determine or generate at least one predicted pixel based on at least one first image block feature determined or generated by convolving at least one reference information of the current block using at least one convolutional layer of a neural network model;

[0025] Determine or generate at least one predicted pixel based on at least one second image block feature determined or generated by performing full-connection processing on at least one reference information of the current block using at least one fully-connected layer of a neural network model;

[0026] Determine or generate at least one predicted pixel based on a fusion feature determined or generated by fusing at least one third image block feature corresponding to at least one reference information of the current block using a neural network model;

[0027] At least one predicted pixel is determined or generated based on at least one fourth image block feature determined or generated by performing convolution processing on a fused feature corresponding to at least one reference information of the current block using a neural network model.

[0028] Optionally, the first parameter includes a first prediction mode, and step S10 includes:

[0029] The current block is processed according to the first prediction mode to determine or generate at least one predicted pixel.

[0030] Optionally, the processing method further includes at least one of the following:

[0031] At least one predicted pixel is a first predicted pixel;

[0032] Determining or generating a first prediction mode according to at least one first prediction pixel;

[0033] According to the first prediction mode, prediction processing is performed on the current block to obtain at least one second predicted pixel.

[0034] Optionally, the processing method further includes at least one of the following:

[0035] Determine a residual corresponding to at least one second predicted pixel according to a syntax element obtained from a bitstream;

[0036] determining or generating a reconstructed pixel based on the residual and at least one second predicted pixel;

[0037] A corresponding residual is determined or generated according to at least one original pixel and at least one second predicted pixel corresponding to the current block.

[0038] The present application also provides a processing device, including: a memory and a processor, wherein a processing program is stored in the memory, and when the processing program is executed by the processor, the steps of any of the above processing methods are implemented.

[0039] The present application also provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the processing methods described above are implemented.

[0040] As described above, the processing method of the present application can be applied to a processing device, including: determining or generating at least one predicted pixel based on a first parameter. Through the technical solution of the present application, it is possible to determine or generate at least one predicted pixel based on the first parameter, thereby improving the accuracy of pixel prediction and, in turn, improving the encoding and decoding quality during the video encoding and / or decoding process. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for describing the embodiments. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without inventive work.

[0042] FIG1 is a schematic diagram of the hardware structure of a mobile terminal for implementing various embodiments of the present application;

[0043] FIG2 is a diagram of a communication network system architecture provided by an embodiment of the present application;

[0044] FIG3 is a schematic diagram of the hardware structure of a controller 140 provided in this application;

[0045] FIG4 is a schematic diagram of the hardware structure of a network node 150 provided in this application;

[0046] FIG5 is a schematic flow chart of a processing method according to the first embodiment;

[0047] FIG6 is a schematic diagram of an encoding process of an encoder in an image processing method according to the first embodiment;

[0048] 7 is a schematic diagram of a decoding process of a decoder in an image processing method according to the first embodiment;

[0049] FIG8 is a first schematic diagram of a reference area according to the second embodiment;

[0050] FIG9 is a second schematic diagram of a reference area according to the second embodiment;

[0051] FIG10 is a third schematic diagram of a reference area according to the second embodiment;

[0052] FIG11 is a schematic structural diagram of a convolutional layer-based neural network model according to the third embodiment;

[0053] FIG12 is a structural diagram 1 of a neural network model based on a hybrid convolutional and fully connected layer according to the third embodiment;

[0054] FIG13 is a second structural diagram of a neural network model based on a hybrid convolutional and fully connected layer according to the third embodiment;

[0055] FIG14 is a schematic diagram of a processing module of a processing device.

[0056] The purpose of this application, its features, and advantages will be further described in conjunction with the embodiments and with reference to the accompanying drawings. The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and the accompanying text are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of this application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0057] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0058] It should be noted that, in this document, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0059] It should be understood that although the terms "first," "second," "third," etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are used solely to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the term "if," as used herein, may be interpreted as "upon," "when," or "in response to a determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms "comprising" and "including" indicate the presence of the recited features, steps, operations, elements, components, items, types, and / or groups, but do not preclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, types, and / or groups. The terms "or," "and / or," "including at least one of the following," etc., as used herein, may be interpreted as inclusive, meaning any one or any combination. For example, “comprising at least one of the following: A, B, C” means “any of the following: A; B; C; A and B; A and C; B and C; A and B and C”; and for another example, “A, B or C” or “A, B and / or C” means “any of the following: A; B; C; A and B; A and C; B and C; A and B and C”. An exception to this definition will occur only when a combination of elements, functions, steps or operations are inherently mutually exclusive in some manner.

[0060] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and they can be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0061] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0062] It should be noted that, in this article, step codes such as S10 are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence.

[0063] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0064] In the subsequent description, the use of suffixes such as "module", "component" or "unit" to represent elements is only for the purpose of facilitating the description of the present application and has no specific meaning. Therefore, "module", "component" or "unit" can be used interchangeably.

[0065] The processing device may be implemented in various forms. For example, the processing device described in this application may include processing devices such as mobile phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, and fixed terminals such as digital TVs and desktop computers.

[0066] The subsequent description will be made by taking a mobile terminal as an example. It will be understood by those skilled in the art that, in addition to components specifically used for mobile purposes, the configuration according to the embodiments of the present application can also be applied to fixed-type terminals.

[0067] Please refer to Figure 1, which is a schematic diagram of the hardware structure of a mobile terminal for implementing various embodiments of the present application. The mobile terminal 100 may include components such as an RF (Radio Frequency) unit 101, a WiFi module 102, an audio output unit 103, an A / V (Audio / Video) input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, a processor 110, and a power supply 111. Those skilled in the art will understand that the mobile terminal structure shown in Figure 1 does not limit the mobile terminal. The mobile terminal may include more or fewer components than shown, or may combine certain components, or arrange the components differently.

[0068] The following is a detailed introduction to the various components of the mobile terminal in conjunction with Figure 1:

[0069] The RF unit 101 can be used to send and receive information or receive signals during calls. Specifically, it receives downlink information from the base station and transmits it to the processor 110 for processing; and / or sends uplink data to the base station. Typically, the RF unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, and more. Furthermore, the RF unit 101 can communicate with the network and other devices via wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution), TDD-LTE (Time Division Duplexing-Long Term Evolution), 5G and 6G, etc.

[0070] WiFi is a short-range wireless transmission technology. A mobile terminal, through WiFi module 102, enables users to send and receive emails, browse web pages, and access streaming media, providing wireless broadband Internet access. Although FIG1 illustrates WiFi module 102, it is understood that it is not a required component of the mobile terminal and can be omitted as needed without altering the essence of the invention.

[0071] The audio output unit 103 can convert audio data received by the RF unit 101 or the WiFi module 102 or stored in the memory 109 into an audio signal and output it as sound when the mobile terminal 100 is in a call signal reception mode, a talk mode, a recording mode, a voice recognition mode, a broadcast reception mode, or the like. Furthermore, the audio output unit 103 can also provide audio output related to a specific function performed by the mobile terminal 100 (e.g., a call signal reception sound, a message reception sound, etc.). The audio output unit 103 may include a speaker, a buzzer, or the like.

[0072] The A / V input unit 104 is used to receive audio or video signals. The A / V input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos captured by an image capture device (e.g., a camera) in video capture mode or image capture mode. The processed image frames may be displayed on the display unit 106. The image frames processed by the GPU 1041 may be stored in the memory 109 (or other storage medium) or transmitted via the RF unit 101 or the WiFi module 102. The microphone 1042 may receive sound (audio data) in operating modes such as a phone call mode, a recording mode, and a voice recognition mode, and may process such sound into audio data. In the phone call mode, the processed audio (voice) data may be converted into a format that can be transmitted to a mobile communication base station via the RF unit 101. The microphone 1042 may implement various types of noise cancellation (or suppression) algorithms to eliminate (or suppress) noise or interference generated during the reception and transmission of audio signals.

[0073] The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Optionally, the light sensor includes an ambient light sensor and a proximity sensor. Optionally, the ambient light sensor can adjust the brightness of the display panel 1061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 1061 and / or the backlight when the mobile terminal 100 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0074] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0075] The user input unit 107 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile terminal. Optionally, the user input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any other suitable object or accessory on or near the touch panel 1071) and drive the corresponding connection device according to a pre-set program. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Optionally, the touch detection device detects the user's touch direction and detects the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 110. It can also receive commands sent by the processor 110 and execute them. In addition, the touch panel 1071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1071, the user input unit 107 may further include other input devices 1072. Optionally, the other input devices 1072 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power keys, etc.), a trackball, a mouse, a joystick, etc., and the specifics are not limited here.

[0076] Optionally, the touch panel 1071 may overlay the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the processor 110 to determine the type of touch event. The processor 110 then provides a corresponding visual output on the display panel 1061 based on the type of touch event. Although in FIG1 , the touch panel 1071 and the display panel 1061 are shown as two separate components to implement the input and output functions of the mobile terminal, in some embodiments, the touch panel 1071 and the display panel 1061 may be integrated to implement the input and output functions of the mobile terminal, which is not limited to this specific embodiment.

[0077] The interface unit 108 serves as an interface through which at least one external device can be connected to the mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 108 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the mobile terminal 100 or may be used to transmit data between the mobile terminal 100 and an external device.

[0078] Memory 109 can be used to store software programs and various data. Memory 109 may primarily include a program storage area and a data storage area. Optionally, the program storage area may store an operating system and at least one application required for a function (such as a sound playback function or an image playback function); the data storage area may store data generated based on the use of the mobile phone (such as audio data, a phone book, etc.). Furthermore, memory 109 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0079] Processor 110 is the control center of the mobile terminal, connecting all components of the mobile terminal using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 109 and accessing data stored in memory 109, it executes various functions of the mobile terminal and processes data, thereby providing overall monitoring of the mobile terminal. Processor 110 may include one or more processing units; preferably, processor 110 may integrate an application processor and a modem processor. Optionally, the application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 110.

[0080] The mobile terminal 100 may also include a power supply 111 (such as a battery) for supplying power to various components. Preferably, the power supply 111 may be logically connected to the processor 110 through a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0081] Although not shown in FIG. 1 , the mobile terminal 100 may further include a Bluetooth module, etc., which will not be described in detail here.

[0082] To facilitate understanding of the embodiments of the present application, the communication network system on which the mobile terminal of the present application is based is described below.

[0083] Please refer to Figure 2, which is a communication network system architecture diagram provided in an embodiment of the present application. The communication network system is an LTE system of universal mobile communication technology. The LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203 and an operator's IP service 204, which are connected in sequence.

[0084] Optionally, UE201 may be the above-mentioned terminal 100, which will not be described in detail here.

[0085] E-UTRAN 202 includes eNodeB 2021 and other eNodeBs 2022 . Optionally, eNodeB 2021 may be connected to other eNodeBs 2022 via a backhaul (eg, an X2 interface). eNodeB 2021 is connected to EPC 203 , and eNodeB 2021 may provide access from UE 201 to EPC 203 .

[0086] EPC 203 may include an MME (Mobility Management Entity) 2031, an HSS (Home Subscriber Server) 2032, other MMEs 2033, an SGW (Serving Gate Way) 2034, a PGW (PDN Gate Way) 2035, and a PCRF (Policy and Charging Rules Function) 2036. MME 2031 is optionally a control node that processes signaling between UE 201 and EPC 203, providing bearer and connection management. HSS 2032 provides registers for managing functions such as the Home Location Register (not shown) and stores user-specific information such as service features and data rates. All user data can be sent through SGW2034, PGW2035 can provide IP address allocation and other functions for UE 201, PCRF2036 is the policy and charging control policy decision point for service data flow and IP bearer resources, and it selects and provides available policy and charging control decisions for the policy and charging execution function unit (not shown in the figure).

[0087] The IP service 204 may include the Internet, an intranet, an IMS (IP Multimedia Subsystem), or other IP services.

[0088] Although the above introduction takes the LTE system as an example, those skilled in the art should know that this application is not only applicable to the LTE system, but can also be applied to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA, 5G and future new network systems (such as 6G), etc., which are not limited here.

[0089] FIG3 is a schematic diagram of the hardware structure of a controller 140 provided in this application. The controller 140 includes a memory 1401 and a processor 1402. The memory 1401 is used to store program instructions, and the processor 1402 is used to call the program instructions in the memory 1401 to execute the steps performed by the controller in the first embodiment of the above method. The implementation principles and beneficial effects are similar and will not be repeated here.

[0090] Optionally, the controller further includes a communication interface 1403, which can be connected to the processor 1402 via a bus 1404. The processor 1402 can control the communication interface 1403 to implement the receiving and sending functions of the controller 140.

[0091] Figure 4 is a schematic diagram of the hardware structure of a network node 150 provided in this application. Network node 150 includes: a memory 1501 and a processor 1502. Memory 1501 is used to store program instructions, and processor 1502 is used to call the program instructions in memory 1501 to execute the steps performed by the first node in the first embodiment of the above method. The implementation principles and beneficial effects are similar and will not be repeated here.

[0092] Optionally, the controller further includes a communication interface 1503, which can be connected to the processor 1502 via a bus 1504. The processor 1502 can control the communication interface 1503 to implement the receiving and sending functions of the network node 150.

[0093] The integrated modules implemented in the form of software function modules can be stored in a computer-readable storage medium. The software function modules stored in a storage medium include a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute some of the steps of the methods of various embodiments of the present application.

[0094] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive solid state disk, SSD), etc.

[0095] Based on the above-mentioned mobile terminal hardware structure and communication network system, various embodiments of the present application are proposed.

[0096] First embodiment

[0097] 5 , which is a flow chart of a processing method according to a first embodiment, the processing method of the embodiment of the present application can be applied to a processing device, including:

[0098] Step S10 : determining or generating at least one predicted pixel according to a first parameter.

[0099] In this embodiment, the processing device can be a smart terminal, such as a mobile phone, a computer, etc., or a server, such as a local server or a cloud server. In this embodiment and this application, the processing device is mainly described as a smart terminal.

[0100] Optionally, the technical solution of this embodiment can be applied to the fields of image coding and decoding, video coding and decoding, hardware video coding and decoding, dedicated circuit video coding and decoding, real-time video coding and decoding, and so on.

[0101] Optionally, the predicted pixel may be a predicted value of at least one pixel in the current block determined or generated according to the first parameter.

[0102] Optionally, the processing device may obtain video image data from a video source, segment each frame of the video image data to obtain multiple image blocks, and determine an image block to be predicted among the multiple image blocks as the current block.

[0103] Optionally, the current block may be an image block to be predicted in an encoder and / or a decoder.

[0104] Optionally, the prediction block may be further interpolated to obtain an interpolated prediction block.

[0105] Optionally, the first parameter may include reference information, a first model, model parameters, a first prediction mode, weight information, a reference area, a reference block, a reference frame, a motion vector, a reference frame index, partition information, filtering parameters, etc.

[0106] Optionally, the weight information may be information about the proportion of each reference block that contributes to the final prediction result when processing the current block in combination with the reference blocks of the current block. For example, the weight information may be a weight coefficient; the weight information may also be information indicating the proportion of different elements (e.g., predicted values) in the result or decision, such as a weight vector or weight matrix containing the weight coefficient.

[0107] Alternatively, the reference block can be a pixel block corresponding to the current block extracted from a reference frame (also called a reference image) or the current frame (also called the current image). The reference block can have the same size as the current block and can serve as a prediction block for generating the current block. By using reference blocks, the encoder can exploit temporal or spatial correlations to reduce redundant information in the current frame, thereby achieving efficient compression.

[0108] Optionally, at least one reference block of the current block is determined or obtained according to an index obtained from the bitstream.

[0109] Optionally, a reference frame refers to a frame (also called an image) that has been encoded and reconstructed, which can be a forward reference frame (i.e., a frame / image that is located before the current frame in the playback order) or a backward reference frame (i.e., a frame / image that is located after the current frame in the playback order).

[0110] Optionally, the inter-frame prediction mode based on the first parameter in the embodiment of the present application can be used as an independent geometric partitioning mode to compete with other geometric partitioning modes in the encoder and / or decoder. Optionally, the final prediction mode can be determined or obtained based on the rate-distortion cost of each geometric partitioning mode applied to the current block. Optionally, when the final prediction mode is the inter-frame prediction mode based on the first parameter provided in this embodiment, the above-mentioned step S10 can be performed.

[0111] Optionally, the first parameter includes a first prediction mode, and step S10 includes:

[0112] The current block is processed according to the first prediction mode to determine or generate at least one predicted pixel.

[0113] Optionally, the first prediction mode is the prediction mode of an embodiment of the present application, and can compete with other modes, for example, a geometric partitioning mode (GPM); when the prediction mode is determined to be the first prediction mode according to the first parameter, at least one predicted pixel is determined or generated according to other contents in the first parameter.

[0114] Optionally, GPM mode is a prediction mode used by the next-generation video compression standard (Versatile Video Coding, VVC) for the boundaries of moving objects in an image. Compared to other prediction modes, GPM mode can more finely delineate the boundaries of moving objects in an image. By aligning the dividing line with the boundary of the moving object, the edge coding unit (Code Unit) of the moving object is divided into non-rectangular sub-coding units for unidirectional prediction, thereby obtaining the predicted value of the entire coding unit.

[0115] Optionally, the first parameter includes: a first prediction mode, reference information and a first model; according to the first prediction mode, the reference information is input into the first model to determine or generate at least one predicted pixel.

[0116] The prediction mode used in the embodiment of the present application can compete with the prediction modes in other encoders and / or decoders. For example, the final prediction mode is determined or obtained based on the rate-distortion cost of each mode applied to the current block. When the final prediction mode is the first prediction mode, step S10 is performed to increase the flexibility of model selection during the encoding and decoding process and improve the prediction effect.

[0117] Optionally, referring to Figure 6, when the processing device is an encoder on the encoding side, the encoder can receive video data input from a video source, for example, the encoder receives a video image from a video source, determines an image to be predicted in the video image, divides the image to be predicted into multiple image blocks, and utilizes the temporal and / or spatial correlation between the video images to perform prediction processing on each of the multiple image blocks, including intra-frame prediction processing and / or inter-frame prediction processing, and the intra-frame prediction processing and / or inter-frame prediction processing respectively include multiple prediction modes. For these prediction modes, the encoder uses, for example, rate-distortion cost to determine the prediction mode ultimately adopted by each of the multiple image blocks, for example, calculating the rate-distortion cost corresponding to each prediction mode or the rate-distortion cost of a combination of several prediction modes to determine the minimum rate-distortion cost from the multiple rate-distortion costs. The prediction mode or combination of prediction modes corresponding to the minimum rate-distortion cost is the prediction mode ultimately adopted by the image block.

[0118] Optionally, referring to FIG6 , after determining or generating at least one predicted pixel based on step S10, a residual between the predicted pixel and the original pixel of the current block can be calculated; the residual can be transformed and quantized, and then encoded by an entropy encoder to form an encoded bitstream. Furthermore, the encoded bitstream can also include prediction parameters corresponding to the determined prediction mode and related side information. Optionally, the prediction parameters are entropy encoded and packaged into the encoded bitstream. Optionally, the prediction parameters include information indicating the prediction mode. Optionally, the transformed and quantized residual block can be inverse quantized and inverse transformed, and then added to corresponding prediction data (e.g., predicted pixels, prediction blocks, etc.) obtained using the prediction mode to obtain reconstructed data (e.g., reconstructed pixels and / or reconstructed blocks). After obtaining the reconstructed data, a loop filtering module performs loop filtering on the reconstructed data according to filter control parameters to reduce distortion. After loop filtering, the loop filtered reconstructed data is stored in a coded image buffer.

[0119] Optionally, referring to FIG7 , when the processing device is a decoder on the decoding side, after receiving the encoded bitstream, the decoder's entropy decoding unit parses and decodes the encoded bitstream to obtain transform coefficients. The decoder's inverse transform unit and inverse quantization unit perform inverse transform and inverse quantization on the transform coefficients to obtain a residual block. Optionally, the decoder's entropy decoding unit parses and decodes the encoded bitstream to obtain prediction data, such as prediction parameters and related auxiliary information. The decoder's prediction processing unit performs prediction processing using the prediction parameters to determine predicted pixels and / or prediction blocks corresponding to the residual. Optionally, the prediction processing includes intra-frame prediction processing and / or inter-frame prediction processing, and the intra-frame prediction processing and / or inter-frame prediction processing each include a combination of at least one prediction mode from a plurality of prediction modes. When the prediction parameter indicates that the corresponding prediction mode is the prediction mode provided in this embodiment, the prediction mode is the prediction mode corresponding to the obtained residual block.

[0120] Optionally, referring to Figure 7, when the prediction mode in this embodiment is adopted, the above step S10 is performed. After obtaining the prediction data of the current block (for example, predicted pixels, predicted blocks), the obtained residual and the corresponding prediction data are added to obtain reconstructed data (for example, reconstructed pixels, reconstructed blocks, etc.). The loop filtering unit in the decoder performs loop filtering on the reconstructed blocks according to the filter control data to reduce distortion and improve video quality. The reconstructed data after loop filtering is further combined into a decoded image and stored in a decoded image buffer or output as decoded video data.

[0121] Optionally, a prediction block for the current block can be determined or generated based on the predicted pixels, and a residual block between the predicted block and the current block can be calculated. The residual block is then transformed and quantized, and then encoded by an entropy encoder to form an encoded bitstream. Optionally, the predicted pixels can be further processed accordingly, for example, using another model. Based on the predicted pixels obtained after the processing, a prediction block is determined or generated, and the step of calculating the residual block between the predicted block and the current block and subsequent steps are performed.

[0122] Optionally, in the first prediction mode, a more complex interpolation filter (e.g., a high-order polynomial filter, an adaptive interpolation filter, etc.) can be used to process the predicted pixels to determine or generate a prediction block to improve the matching between the prediction block and the original image block (i.e., the current block), especially when processing scenes with complex motion or edges.

[0123] Optionally, in the first prediction mode, the predicted pixels can be further predicted using an angular prediction mode; by taking into account the directional characteristics of the image content (such as horizontal, vertical, diagonal, etc.), the direction that best reflects these characteristics is selected for prediction to obtain a predicted block. Optionally, the inter-frame prediction mode based on the first parameter of the embodiment of the present application can be fused with the angular prediction mode to generate a fused prediction value and a corresponding prediction block.

[0124] Optionally, in conventional intra-frame prediction and / or inter-frame prediction processes, pixel prediction results are often unsatisfactory, leading to poor encoding and decoding quality during video encoding and / or decoding. In this embodiment, by determining or generating at least one predicted pixel based on the first parameter, the accuracy of pixel prediction can be effectively improved, thereby improving the encoding and decoding quality during video encoding and / or decoding.

[0125] Second embodiment

[0126] Based on the first embodiment, a second embodiment is proposed.

[0127] In this embodiment, the first parameter includes: at least one reference information of the current block and a first model.

[0128] Optionally, reference information is data and / or parameters related to determining or generating at least one predicted pixel of the current block. Reference information can help the encoder and / or decoder construct a predicted pixel that is as similar as possible to a pixel in the original image, thereby constructing a predicted block, thereby improving prediction effectiveness and encoding and decoding quality during video encoding and / or decoding. Reference information can optionally include: a reference region, a reference block, a reference frame, a motion vector, a reference frame index, partition information, weight information, filtering parameters, etc.

[0129] Optionally, the first model may include: a neural network model, a regression model, a matrix-based mathematical model, a lookup table-based mathematical model, a normalization operation-based mathematical model, etc. The first model can more deeply learn and capture complex patterns and features in the image (i.e., the current block), such as textures and edges, thereby generating more accurate prediction results, thereby effectively reducing redundancy and further reducing the amount of data required to represent the residual, thereby reducing the overall bit rate and improving the encoding and decoding quality during the video encoding and / or decoding process.

[0130] Optionally, at least one reference information of the current block can be input into the first model, and the first model can more deeply learn and capture the association between the at least one reference information and the current block, and determine or generate at least one more accurate predicted pixel, thereby improving the accuracy of pixel prediction.

[0131] Optionally, the at least one reference information of the current block includes pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block of the current block.

[0132] Optionally, the reference area can be an image area used to determine or obtain at least one predicted pixel of the current block, and can be an image area adjacent to the current block, or an image area non-adjacent to the current block. Optionally, the reference area can be an area composed of encoded and reconstructed pixels or blocks from the same frame and / or different frames. Optionally, the reference area can include at least one of pixels, image blocks, and image areas used to determine or generate at least one predicted pixel of the current block. Since there is a high degree of correlation (e.g., temporal correlation, spatial correlation, etc.) between the reference area and the current block, the embodiments of the present application can significantly improve the pixel prediction accuracy of the current block by utilizing these associations.

[0133] Alternatively, the reference block can be a pixel block corresponding to the current block extracted from a reference frame (also called a reference image) or the current frame (also called the current image). The reference block can have the same size as the current block and can serve as a prediction block for generating the current block. By using reference blocks, the encoder can exploit temporal or spatial correlations to reduce redundant information in the current frame, thereby achieving efficient compression.

[0134] Optionally, pixels of at least one reference area of ​​the current block are input into the first model to determine or generate at least one predicted pixel.

[0135] Optionally, pixels of at least one reference block of the current block are input into a first model to determine or generate at least one predicted pixel.

[0136] Optionally, pixels of at least one reference area of ​​the current block and pixels of at least one reference block are input into the first model to determine or generate at least one predicted pixel.

[0137] Optionally, the first model includes a first sub-model and a second sub-model; pixels of at least one reference area of ​​the current block are input into the first sub-model to determine or generate at least one weight information; at least one weight information and pixels of at least one reference block are input into the second sub-model to determine or generate at least one predicted pixel.

[0138] Optionally, the weight information may be information about the proportion of each reference block's contribution to the final prediction result during the process of processing the current block in combination with the reference blocks of the current block. The weight information may reflect the reliability and / or relevance of each reference block. The higher the reliability and / or relevance of the reference block (e.g., a reference block with a small motion estimation error or a high texture matching degree), the larger the value in the corresponding weight information.

[0139] Optionally, the weight information may include at least one of a weight coefficient, a weight vector, and a weight matrix. A weight coefficient may be a scalar value that represents the weight of a single reference block. A weight vector may be a one-dimensional array containing multiple weight coefficients, each corresponding to a reference block or prediction block. A weight matrix may be a two-dimensional array, where each element in the weight matrix represents a weight coefficient for a pixel at a specific location, which is typically the same size as the current block.

[0140] Optionally, the weight information may be a weight coefficient; the weight information may also be a weight vector and a weight matrix containing weight coefficients, etc., which represent indicative information for measuring the proportion of different elements (for example, predicted values) in the result or decision. Optionally, multiple information or pixel values ​​of the same type can be fused according to the corresponding proportion indicated by the weight information through weighted processing to obtain a corresponding fusion result (for example, predicted pixel).

[0141] Understandably, each partitioning method employed for the current block requires separate signaling of the partitioning method, resulting in high signaling overhead for block partitioning, which in turn limits video compression efficiency. Furthermore, because the boundaries of moving objects are not always straight lines, conventional linear partitioning methods can affect pixel prediction accuracy to a certain extent. Furthermore, the angle and offset of the dividing lines are limited to a series of preset discrete values, further increasing the roughness of the linear partitioning, which in turn affects pixel prediction accuracy.

[0142] Optionally, in an embodiment of the present application, if the processing device is a decoder, the decoder can determine or obtain reference information (e.g., reference area and / or reference block) and a first model from the decoded encoded bit stream, and then directly process the reference information through the first model to determine or generate at least one predicted pixel of the current block.

[0143] Optionally, the first model can automatically select a partitioning method suitable for the current block, or directly output the prediction result without performing block partitioning; and when the input data (i.e., reference information) is the same, the first model can perform the same processing means and give the same output result, i.e., directly determine or generate the predicted pixel, so the bitstream does not need to encode the partitioning method information in the GPM mode, thereby saving the signaling required to transmit the GPM partitioning method. In this way, the implementation method of the present application can effectively reduce the signaling overhead and improve the compression efficiency of the video. And / or, because the first model can automatically select a partitioning method suitable for the current block, or directly output the prediction result without performing block partitioning, the embodiment of the present application can effectively eliminate the impact of linear partitioning on the prediction result of the current block, thereby improving the effect of pixel prediction and the encoding and decoding quality during video encoding and / or decoding.

[0144] Optionally, the division method selected by the first model may include: division by at least one of straight lines, oblique lines, curves, rectangles, circles, ellipses, etc., and the sub-regions obtained by division may be regular matrices and / or irregular matrices.

[0145] Optionally, the target partitioning method may also include: fixed-size partitioning (for example, macroblocks, coding tree units, etc.), adaptive partitioning (for example, quadtree partitioning, binary tree partitioning, ternary tree partitioning, etc.), geometric partitioning mode (for example, 45-degree diagonal partitioning, horizontal partitioning, vertical partitioning, polygonal partitioning, etc.), curve partitioning and at least one of mixed partitioning.

[0146] Optionally, the method for determining or generating at least one reference area includes at least one of Method 1 to Method 6:

[0147] Method 1: determining or obtaining at least one reference area based on at least one of an upper adjacent pixel, an upper non-adjacent pixel, a left adjacent pixel, a left non-adjacent pixel, an upper left adjacent pixel, and an upper left non-adjacent pixel of the current block;

[0148] Optionally, the upper adjacent pixel may be a pixel located above and adjacent to the current block in the same frame image.

[0149] Optionally, the upper non-adjacent pixel may be a pixel in the same frame image that is located above the current block but is not adjacent to the current block.

[0150] Optionally, the left adjacent pixel may be a pixel located on the left side of and adjacent to the current block in the same frame image.

[0151] Optionally, the left non-adjacent pixel may be a pixel located on the left side of the current block but not adjacent to the current block in the same frame image.

[0152] Optionally, the upper left adjacent pixel may be a pixel located in the upper left of the current block and adjacent in the same frame image.

[0153] Optionally, the upper left non-adjacent pixel may be a pixel located in the upper left of the current block but not adjacent to the current block in the same frame image.

[0154] Optionally, at least one of the upper adjacent pixel, the upper non-adjacent pixel, the left adjacent pixel, the left non-adjacent pixel, the upper left adjacent pixel and the upper left non-adjacent pixel may be a reconstructed pixel or a predicted pixel.

[0155] Optionally, at least one of the upper adjacent pixel, the upper non-adjacent pixel, the left adjacent pixel, the left non-adjacent pixel, the upper left adjacent pixel and the upper left non-adjacent pixel can be directly used as a reference area, or at least one acquired pixel can be deduced or calculated to obtain a reference area.

[0156] Optionally, the reference area of ​​the current block may be acquired according to a preset mapping / correspondence rule.

[0157] Optionally, a reference area may be selected from upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels, and upper left non-adjacent pixels of the current block according to a prediction model.

[0158] 8 , the upper adjacent pixels of the current block are determined as reference region A, the left adjacent pixels of the current block are determined as reference region B, and the upper left adjacent pixels of the current block are determined as reference region C. The upper adjacent pixels, the left adjacent pixels, and the upper left adjacent pixels each include multiple pixels.

[0159] Optionally, at least one reference area is determined or obtained based on at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels and upper left non-adjacent pixels of the current block; and at least one predicted pixel is determined or generated based on the pixels of the at least one reference area, the pixels of the at least one reference block and the first model.

[0160] Optionally, if some locations lack valid pixel data, they can be filled with adjacent valid pixels.

[0161] In this embodiment, at least one reference area is determined or generated by introducing intra-frame information of the current block (i.e., at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels, and upper left non-adjacent pixels of the current block). This ensures a close association between the reference area and the current block, and dynamically adjusts the prediction result using the spatial features of the current frame to which the current block belongs, so as to improve the accuracy of pixel prediction and thereby improve the encoding and decoding quality during video encoding and / or decoding.

[0162] Method 2: determining or obtaining at least one reference area according to at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block, and a default block corresponding to the current block;

[0163] Optionally, at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block, and a default block corresponding to the current block is used as at least one reference area.

[0164] Optionally, at least one reference area is determined or obtained according to image block information of at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block, and a default block corresponding to the current block.

[0165] Optionally, the image block information may include at least one of block size, block area, image block attribute and image block type.

[0166] Optionally, the block size includes the width, height, scale, depth, area, resolution, and number of pixels of the block. The image block attributes may include the position and / or image texture of the block. The image block type may include a natural image or a screen content image.

[0167] Optionally, at least one reference area can be determined or obtained based on at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels and upper left non-adjacent pixels of at least one of the neighboring blocks, non-neighboring blocks, co-located blocks, time domain blocks and default blocks corresponding to the current block.

[0168] Optionally, at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels and upper left non-adjacent pixels of at least one of the neighboring blocks, non-neighboring blocks, co-located blocks, time domain blocks and default blocks corresponding to the current block can be used as at least one reference area.

[0169] Optionally, at least one reference area may be determined or obtained based on at least one of the width, height, block size and block area of ​​at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block and a default block corresponding to the current block.

[0170] Optionally, the neighbor block may be a block adjacent to the current block, and may be a block that has been predicted or reconstructed.

[0171] Alternatively, the non-neighbor block may be a block that is not adjacent to the current block and may be a block that has been predicted or reconstructed.

[0172] Optionally, the co-located block may be an image block in a co-located image that has the same position and size as the current block, and the co-located image may be an image in the reference image that is closest to the current image in terms of time.

[0173] Optionally, the time domain block may be a block distinguished from the time domain, such as an image block in another frame before or after the current frame. For example, if video data includes a first frame, a second frame, and a third frame played at the first second, second second, and third second, respectively, and the current block is a block divided from the second frame, then the time domain block corresponding to the current block may be determined to be a corresponding image block in another frame except the second frame.

[0174] Optionally, the default block may be a block set in advance, for example, a block with typical pixel features pre-set by an encoder and / or a decoder.

[0175] Optionally, at least one reference area is determined or obtained based on at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block and a default block corresponding to the current block; and at least one predicted pixel is determined or generated based on the pixels of the at least one reference area, the pixels of the at least one reference block and the first model.

[0176] In this embodiment, at least one reference area is determined or obtained based on at least one of the neighboring blocks, non-neighboring blocks, co-located blocks, time domain blocks and default blocks corresponding to the current block, thereby ensuring a close correlation between the determined or generated reference area and the current block, improving the accuracy of the predicted pixels determined or generated based on the at least one reference area, and thereby improving the encoding and decoding quality during the video encoding and / or decoding process.

[0177] Method three, determining or obtaining at least one reference area according to at least one of the width, height, block size, and block area of ​​the current block;

[0178] Optionally, at least one of pixels, image blocks and image regions in at least one reference region is determined or obtained based on at least one of width, height, block size and block area of ​​the current block.

[0179] Optionally, at least one reference area may be determined or obtained based on the width and height of the current block and a first mapping table. Optionally, the first mapping table may be as shown in Table 1 below:

[0180] Table 1

[0181] Optionally, the height of the at least one reference region is equal to a first preset multiple of the height of the current block. Optionally, the width of the at least one reference region is equal to a second preset multiple of the width of the current block.

[0182] Optionally, the reference area or the image area in the reference area may be an encoded area or a decoded area.

[0183] Alternatively, the coded region or the decoded region may be determined by the position of the upper left corner pixel, the height and width of the coded region or the decoded region. For example, the position of the upper left corner pixel may be N (N is greater than 1) times the height of the image block above and N times the width of the image block to the left of the upper left corner of the current block. The width of the coded region or the decoded region is an integer multiple of the width of the current block, and the height of the coded region or the decoded region is an integer multiple of the height of the current block.

[0184] Alternatively, the reference area may be determined based on the block size of the current block and a second mapping table. Alternatively, the second mapping table may be as shown in Table 2 below:

[0185] Table 2

[0186] Optionally, the block size includes at least one of width, height, scale, depth, area, resolution, and number of pixels of the block. Optionally, X4 to X7 may be preset thresholds corresponding to at least one of width, height, scale, depth, area, resolution, and number of pixels of the block size.

[0187] Alternatively, the reference area may be determined based on the block area of ​​the current block and a third mapping table. Alternatively, the third mapping table may be as shown in Table 3 below:

[0188] Table 3

[0189] Optionally, the position of the reference area can be determined or obtained based on the upper adjacent pixels, left adjacent pixels and upper left adjacent pixels of the current block, and then the size of the reference area can be determined based on the current block width and height and the first mapping table.

[0190] Optionally, the reference area of ​​the current block includes: a first area adjacent to the top of the current block, a second area adjacent to the left of the current block, and a third area adjacent to the top left of the current block. The width of the first area is equal to twice the width of the current block, and the height of the second area is equal to twice the height of the current block. If the width and height of the current block are both greater than or equal to 8, the height of the first area is 8, the width of the second area is 8, and the width and height of the third area are both 8. If either the width or height of the current block is less than 8, the height of the first area is 4, the width of the second area is 4, and the width and height of the third area are both 4.

[0191] Optionally, at least one reference area is determined or obtained based on at least one of the width, height, block size and block area of ​​the current block; and at least one predicted pixel is determined or generated based on the pixels of the at least one reference area, the pixels of the at least one reference block and the first model.

[0192] In this embodiment, by determining or obtaining at least one reference area based on at least one of the width, height, block size and block area of ​​the current block, a close correlation between the determined or generated reference area and the current block is ensured, thereby improving the accuracy of the predicted pixels determined or generated based on the at least one reference area, thereby improving the encoding and decoding quality during the video encoding and / or decoding process.

[0193] Method 4, determining or obtaining at least one reference area based on a candidate block determined or obtained based on a candidate motion vector or a candidate block vector of a current block;

[0194] Optionally, a candidate block may be determined or obtained according to a plurality of candidate motion vectors or a plurality of candidate block vectors in a candidate list of the current block, and used as at least one reference area.

[0195] Optionally, a candidate block can be determined or obtained based on a candidate motion vector or a candidate block vector of the current block; and at least one reference area can be determined or obtained based on at least one of upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels, and upper left non-adjacent pixels of the candidate block. The specific implementation process can refer to the solution in the above-mentioned method 1, that is, the current block in method 1 can be replaced with the candidate block, and the details are not repeated here.

[0196] Optionally, a candidate block can be determined or obtained based on the candidate motion vector or candidate block vector of the current block; and at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels and upper left non-adjacent pixels of the candidate block is used as at least one reference area.

[0197] Optionally, a candidate block can be determined or obtained based on a candidate motion vector or candidate block vector of the current block; and at least one reference region can be determined or obtained based on at least one of the width, height, block size, and block area of ​​the candidate block. The specific implementation process can refer to the solution in Method 3 above, that is, the current block in Method 3 can be replaced with the candidate block, and the details are not repeated here.

[0198] Optionally, a candidate block can be determined or obtained based on a candidate motion vector or candidate block vector of the current block; and at least one reference region can be determined or obtained based on at least one of a neighboring block, a non-neighboring block, a co-located block, a temporal block, and a default block corresponding to the candidate block. The specific implementation process can refer to the solution in the above-mentioned method 2, that is, the current block in method 2 can be replaced with the candidate block, and the details are not repeated here.

[0199] Optionally, a candidate block can be determined or obtained based on a candidate motion vector or a candidate block vector of the current block; and at least one of a neighbor block, a non-neighbor block, a co-located block, a time domain block and a default block corresponding to the candidate block is used as at least one reference area.

[0200] Optionally, referring to Figure 9, the best matching block is determined or obtained from the current image based on the best block vector (BV) of the current block; the upper adjacent pixels of the best matching block are determined as the reference area A; the left adjacent pixels of the best matching block are determined as the reference area B; and the upper left adjacent pixels of the best matching block are determined as the reference area C.

[0201] Optionally, referring to Figure 10, a candidate block is determined or obtained from a reference image based on the motion vector (MV) of the current block; the upper adjacent pixels of the candidate block are determined as a reference area A; the left adjacent pixels of the candidate block are determined as a reference area B; and the upper left adjacent pixels of the candidate block are determined as a reference area C.

[0202] Optionally, at least one reference area is determined or obtained based on the candidate block determined or obtained according to the candidate motion vector or candidate block vector of the current block; and at least one predicted pixel is determined or generated based on the pixels of the at least one reference area, the pixels of the at least one reference block and the first model.

[0203] In this embodiment, the candidate block determined or obtained based on the candidate motion vector or candidate block vector of the current block ensures a close association between the determined or generated reference area and the current block, and improves the accuracy of the predicted pixels determined or generated based on at least one reference area, thereby improving the encoding and decoding quality during the video encoding and / or decoding process.

[0204] Method 5: If the first information of the current block meets the first condition, the reference area is the first reference area;

[0205] Optionally, the first information may be at least one of flag information of the current block, syntax elements, indication information, an index of a list, an aspect ratio of the current block, a width and height value range, and an area value range.

[0206] Optionally, flag information can be used to indicate a specific state or option, for example, to control the behavior of an encoder or decoder.

[0207] Optionally, the syntax element refers to a series of parameters and data structures defined in a video coding standard, and is used to describe specific coding information of video frames and blocks.

[0208] Optionally, the indication information is information used to indicate a certain state or condition, and may include flag information or more complex syntax elements.

[0209] Optionally, an index in a list refers to an integer value used to identify a specific item in a list. These lists may contain candidate motion vectors, reference frame indices, transform coefficients, etc. The index allows for quick lookup and access to a specific item in the list.

[0210] Optionally, the first information may be at least one of methods one to four, that is, the first information may be determining or obtaining at least one reference area based on at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels and upper left non-adjacent pixels of the current block; may be determining or obtaining at least one reference area based on at least one of the neighboring blocks, non-neighboring blocks, co-located blocks, time domain blocks and default blocks corresponding to the current block; may be determining or obtaining at least one reference area based on at least one of the width, height, block size and block area of ​​the current block; may be determining or obtaining at least one reference area based on a candidate block determined or obtained according to the candidate motion vector or candidate block vector of the current block.

[0211] Optionally, the first condition may be a condition set in advance and may be set according to user needs. The first condition is not fixed and may be adaptively adjusted according to different scenarios.

[0212] Optionally, if the processing device detects that the current block meets the first condition, it may select the first information for calculation to determine or obtain at least one reference area.

[0213] Optionally, if the first information of the current block satisfies the first condition, the reference area is the first reference area; and at least one predicted pixel is determined or generated based on pixels of at least one reference area, pixels of at least one reference block, and the first model.

[0214] In this embodiment, by selecting a reference area based on the first information, the similarity between the reference area and the current block (for example, the similarity of texture features, the similarity of pixel features, etc.) can be improved, thereby making the predicted pixels determined or generated based on the reference area with high similarity to the current block more accurate, and further improving the encoding and decoding quality during the video encoding and / or decoding process.

[0215] Method six: if the first information of the current block does not meet the first condition, the reference area is the second reference area.

[0216] Optionally, the first reference region and the second reference region are different.

[0217] Optionally, the first reference area may be a portion of at least one of a left adjacent area, an upper adjacent area, and an upper left adjacent area of ​​the current block, and the second reference area may be an area of ​​the above areas different from the first reference area.

[0218] Optionally, if the first information of the current block does not meet the first condition, the reference area is the second reference area; and at least one predicted pixel is determined or generated based on pixels of at least one reference area, pixels of at least one reference block and the first model.

[0219] In this embodiment, by selecting a reference area based on the first information, the similarity between the reference area and the current block (for example, the similarity of texture features, the similarity of pixel features, etc.) can be improved, so that the predicted pixels determined or generated based on the reference area with high similarity to the current block have higher accuracy, thereby improving the encoding and decoding quality during the video encoding and / or decoding process.

[0220] Third embodiment

[0221] Based on any of the above embodiments, a third embodiment is proposed.

[0222] In this embodiment, the first model includes at least one of the seventh to eleventh methods:

[0223] Method seven, neural network model;

[0224] Optionally, the first model includes a neural network model. A neural network model is a computational model that mimics the way neurons in the human brain process information. It performs various tasks by learning complex patterns and features in data. A neural network model consists of multiple layers of neurons, including an input layer, at least one hidden layer, and an output layer. Each neuron receives an input signal, performs a weighted sum, and then generates an output through a nonlinear activation function. The weights connecting these neurons determine the extent to which the input signal affects the output, while the bias is used to adjust the activation threshold of the neuron. Activation functions, such as ReLU, Sigmoid, or Tanh, give neural networks the ability to model nonlinearly, enabling them to solve problems that linear models cannot solve.

[0225] Optionally, the reference information is input into the neural network model, and the association relationship between the current block and its reference information learned by the neural network model is fully utilized to determine or generate at least one predicted pixel.

[0226] Optionally, pixels of at least one reference area of ​​the current block and pixels of at least one reference block of the current block are input into a neural network model to determine or generate at least one predicted pixel.

[0227] Optionally, the neural network model may include: an input layer, a hidden layer, a dropout layer, and an output layer.

[0228] Optionally, the input layer is the first layer to receive input data. In this layer, each node (or neuron) typically represents a feature of the input data. Hidden layers are located between the input and output layers. There can be multiple hidden layers, and these layers process the input data in a weighted manner. Hidden layers include at least one of the following: fully connected layers, convolutional layers, pooling layers, recurrent layers, and dropout layers. A fully connected layer (Dense or Fully Connected Layer): Each neuron is connected to every neuron in the previous layer. A convolutional layer (Pooling Layer): Used for extracting local features from the input data and is commonly used in image processing. A recurrent layer (Recurrent Layer): Used for downsampling to reduce the amount of data and computation. A recurrent layer (Recurrent Layer): Such as an LSTM or GRU, processes sequential data. A dropout layer (Dropout Layer): Used to randomly "drop out" (i.e., temporarily set the output to zero) a portion of neurons during model training. This reduces the complex co-adaptive relationships between neurons, improves the model's generalization ability, and reduces the risk of overfitting. Dropout layers can be added between multiple fully connected layers to prevent neurons in these layers from becoming overly dependent on the specific outputs of the previous layer, thereby enhancing the model's generalization capabilities. Dropout layers can be added between convolutional and fully connected layers to help reduce overfitting introduced by the fully connected layers. The output layer is the layer that generates the final prediction results.

[0229] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0230] In this embodiment, the neural network model possesses powerful nonlinear modeling capabilities, allowing it to automatically learn complex mapping relationships between inputs and outputs from large amounts of data. Therefore, the neural network can identify and utilize subtle connections between the current block and its reference information. For example, it can effectively capture temporal correlations between frames and spatial correlations within frames, understanding the continuity and change patterns between adjacent regions in different frames or within the same frame. Furthermore, it can perceive the consistency of local features such as edges and textures, even if these features are displaced or deformed between frames. Through multi-level feature extraction, it ensures that the prediction results are not only accurate in local details but also consistent with the actual content in terms of overall layout, thereby improving the accuracy of pixel prediction from multiple dimensions. Furthermore, because the predicted pixels can be directly determined or generated by the neural network model, the bitstream does not need to encode the partitioning method information, thereby saving the signaling required to transmit the GPM partitioning method. As a result, the embodiments of the present application can effectively reduce signaling overhead and improve video compression efficiency. And / or, the neural network model can automatically select a division method suitable for the current block, or directly output the prediction result without performing block division; therefore, the embodiment of the present application can also effectively eliminate the impact of linear division on the prediction result of the current block, improve the pixel prediction effect, and the encoding and decoding quality during the video encoding and / or decoding process.

[0231] Optionally, a first neural network model may be determined or obtained from a set of neural network models; and at least one predicted pixel may be determined or generated based on the first neural network model and pixels of at least one reference region of the current block and / or pixels of at least one reference block of the current block. Optionally, the first neural network model may be one or more models.

[0232] Optionally, a first neural network model may be determined or obtained from a set of neural network models using at least one reference region and / or at least one reference block of the current block. A more suitable first neural network model may be selected based on characteristics of the at least one reference region and / or at least one reference block of the current block.

[0233] Optionally, a first neural network model is determined or obtained from a group of neural network models based on at least one reference area and / or at least one reference block and a preset model mapping / correspondence rule.

[0234] Optionally, a set of neural network models are pre-selected and trained, which may include at least one of: a neural network model based on a fully connected layer, a neural network based on a convolutional layer, and a neural network based on a hybrid convolutional and fully connected layer.

[0235] Optionally, step S10 includes at least one of the following methods A to D:

[0236] Method A, determining or generating at least one predicted pixel based on at least one first image block feature determined or generated by convolving at least one reference information of the current block using at least one convolutional layer of a neural network model;

[0237] Optionally, the neural network model includes at least one convolution layer, which can perform at least one convolution process on the input data (i.e., reference information) to determine or generate at least one first image block feature, and determine or generate at least one predicted pixel based on the at least one first image block feature.

[0238] Optionally, at least one predicted pixel is determined or generated based on at least one first image block feature determined or generated by performing convolution processing on pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block using at least one convolution layer of a neural network model.

[0239] Alternatively, the neural network can be a convolutional neural network, i.e., a convolutional neural network (CNN), comprising an input layer, at least one hidden layer, and an output layer. The hidden layer includes a convolutional layer for feature extraction; each convolutional layer may be followed by a pooling layer to reduce spatial size and computational complexity; and each convolutional layer may also be followed by a nonlinear activation function (e.g., ReLU) to introduce nonlinear factors, allowing the network to learn more complex patterns. As the depth of the convolutional neural network increases, the convolutional neural network can automatically learn feature representations from low-level to high-level, improving prediction accuracy.

[0240] Optionally, the at least one first image block feature may be converted into a fixed-length vector via global average pooling (GAP), and at least one predicted pixel may be generated via a fully connected layer or other methods. Optionally, an adaptive weighting mechanism may be used to learn pixel information from the at least one first image block feature using an additional neural network module to determine or obtain the at least one predicted pixel.

[0241] Optionally, referring to FIG11 , the neural network model is a convolutional neural network, comprising: an input layer, multiple hidden layers, and an output layer, wherein the hidden layers comprise: a convolutional layer, an activation function layer (e.g., a LeakyReLU layer), and a linear layer. Pixels of a reference region of a current block and pixels of a reference block are input into the convolutional neural network; the reference region comprises: a reference region A consisting of pixels adjacent to the upper side of the current block, a reference region B consisting of pixels adjacent to the left side of the current block, and a reference region C consisting of pixels adjacent to the upper left side of the current block; the reference blocks comprise: a first reference block and a second reference block. The pixels of the first reference block and the pixels of the second reference block are processed by at least one convolutional layer, and nonlinearly transformed by the ReLU activation function after each convolutional layer to obtain first image block features A after sampling; the pixels of at least one reference region are processed by at least one convolutional layer, and nonlinearly transformed by the ReLU activation function after each convolutional layer to obtain first image block features B; the first image block features A and B are fused to obtain a fused feature; the fused feature is processed by at least one convolutional layer to obtain at least one predicted pixel.

[0242] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0243] Optionally, the first model is determined or obtained based on the width and height of at least one reference area of ​​the current block. Optionally, when the width and height of the reference area are both less than or equal to X1 (e.g., 8), and / or when the height and width of the output predicted pixel are both less than or equal to X1 (e.g., 8), the first model is a convolutional neural network, and the hidden layer structure of the first model is shown in Table 4 below:

[0244] Table 4

[0245] Optionally, when at least one of the width and height of the reference area is greater than X1 (for example, 8), and / or when at least one of the height and width of the output predicted pixel is greater than X1 (for example, 8), the first model is a convolutional neural network, and the hidden layer structure of the first model is shown in Table 5 below:

[0246] Table 5

[0247] In this embodiment, since the connections of the convolutional layer are local and sparse, the number of parameters that the convolutional neural network needs to learn is greatly reduced compared to other neural network models; at the same time, the convolution processing is that the same convolution kernel slides across the entire input space and applies the same weights to detect specific types of features (such as edges, textures, etc.), which can further reduce the number of parameters that the convolutional neural network needs to learn, and also enables the model to more effectively utilize limited data for training, improve the learning efficiency of the model, and help prevent the occurrence of overfitting or underfitting. And / or, by limiting each neuron to only connect to a local area of ​​the input, the convolutional layer can focus on capturing local patterns and structures in the input data, and important information in the image often exists in local areas. Therefore, the accuracy of pixel prediction can be improved through the convolutional neural network, thereby improving the encoding and decoding quality in the video encoding and / or decoding process.

[0248] Mode B, determining or generating at least one predicted pixel based on at least one second image block feature determined or generated by performing full-connection processing on at least one reference information of the current block using at least one fully-connected layer of a neural network model;

[0249] Optionally, the neural network model includes at least one fully connected layer, which can perform at least one fully connected processing on the input data (i.e., reference information) to determine or generate at least one second image block feature, and determine or generate at least one predicted pixel based on the at least one second image block feature.

[0250] Optionally, at least one predicted pixel is determined or generated based on at least one second image block feature determined or generated by performing full-connection processing on pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block using at least one fully-connected layer of a neural network model.

[0251] Optionally, the neural network model can be a fully connected layer-based neural network model, also known as a multilayer perceptron (MLP); this model includes: an input layer, one or more hidden layers (i.e., fully connected layers), and an output layer. Each layer contains a certain number of neurons, and all neurons between two adjacent layers are interconnected, i.e., the layers are "fully connected." Optionally, a dropout layer can be set between at least one group of fully connected layers in the fully connected layer-based neural network model to help reduce overfitting introduced by the fully connected layers.

[0252] Optionally, pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block are fully connected according to at least one fully connected layer of the neural network model, for example, including linear transformation and activation function processing, so as to extract higher-level abstract features from the input and determine or generate at least one second image block feature; and at least one predicted pixel is determined or generated based on the at least one second image block feature.

[0253] Optionally, the at least one second image block feature may be converted into a fixed-length vector via global average pooling (GAP), and at least one predicted pixel may be generated via a fully connected layer or other methods. Optionally, an adaptive weighting mechanism may be employed using an additional neural network module to learn weight information from the at least one second image block feature to determine or obtain the at least one predicted pixel.

[0254] Optionally, at least one first image block feature is determined or generated by performing convolution processing on at least one reference information of the current block according to at least one convolution layer of the neural network model; at least one predicted pixel is determined or generated based on at least one second image block feature that is determined or generated by performing full connection processing on at least one first image block feature using at least one fully connected layer of the neural network model.

[0255] Optionally, at least one second image block feature is determined or generated by performing full-connection processing on at least one first image block feature according to at least one fully-connected layer of a neural network model; at least one predicted pixel is determined or generated based on at least one first image block feature determined or generated by performing convolution processing on at least one second image block feature using at least one convolutional layer of a neural network model.

[0256] Optionally, at least one first image block feature is determined or generated by performing convolution processing on at least one reference information of the current block according to at least one convolution layer of the neural network model; at least one second image block feature is determined or generated by performing full connection processing on at least one reference information of the current block according to at least one fully connected layer of the neural network model; and at least one predicted pixel is determined or generated based on the at least one first image block feature and the at least one second image block feature.

[0257] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0258] In this embodiment, when the reference information includes pixels in a reference area, and the pixels in the reference area are reconstructed reference pixels, since the reconstructed reference pixels provide information already present within the frame, the extraction efficiency requirement is relatively low. Therefore, a fully connected layer can be used to perform more refined feature extraction, thereby improving the accuracy of pixel prediction. Furthermore, the fully connected layer has greater flexibility and can process pixels in reference areas of different sizes (e.g., different numbers of layers), thereby increasing the flexibility of pixel prediction and thereby improving the encoding and decoding quality during the video encoding and / or decoding process.

[0259] Method C, determining or generating at least one predicted pixel based on a fused feature determined or generated by fusing at least one third image block feature corresponding to at least one reference information of the current block using a neural network model;

[0260] Optionally, at least one third image block feature corresponding to at least one reference information of the current block may be determined or obtained through correlation processing. For example, the at least one reference information of the current block may be processed through a neural network model, a regression model, a matrix-based mathematical model, a lookup table-based mathematical model, a normalization-based mathematical model, etc. to determine or obtain the at least one third image block feature.

[0261] Optionally, the third image block feature can be a mathematical representation extracted based on at least one reference information of the current block and capable of describing the content of the current block and / or the reference area; the third image block feature can be a low-level feature (for example, color, brightness, edge, etc.) and / or a high-level feature (for example, texture, shape, semantic information, etc.).

[0262] Optionally, the fused feature is a new feature representation generated by comprehensively processing image block features from multiple sources or different types (i.e., the third image block feature). Fusion can integrate the advantages of each feature to obtain a more comprehensive and accurate descriptor that can better capture the essential characteristics of the image block. Optionally, fusion processing can be performed through weighted summation, cascade connection, multi-layer perceptron (for example, a fully connected layer in a neural network), attention mechanism, adaptive fusion, etc. A single feature may be affected by noise or changes, but the fused feature combines information from multiple sources, therefore, it can offset these adverse factors, making the prediction more stable and reliable, being able to more completely describe the content of the current block, improving the accuracy of the predicted pixels, reducing residuals, and improving the encoding and decoding quality during video encoding and / or decoding.

[0263] Optionally, determining or generating at least one predicted pixel based on the fused features may include directly inputting the fused features into at least one fully connected layer, performing a nonlinear transformation using an activation function, and finally outputting at least one predicted pixel. Furthermore, a convolutional layer may be used to further extract and aggregate spatial features from the fused features, followed by upsampling or deconvolution operations to determine or generate the final at least one predicted pixel.

[0264] Optionally, at least one first image block feature is determined or generated by performing convolution processing on at least one reference information of the current block according to at least one convolution layer of the neural network model; at least one second image block feature is determined or generated by performing full connection processing on at least one reference information of the current block according to at least one fully connected layer of the neural network model; the at least one first image block feature and the at least one second image block feature are used as the third image block feature; the third image block feature is fused to determine or generate a fusion feature; and at least one predicted pixel is determined or generated based on the fusion feature.

[0265] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0266] In this embodiment, by fusing features and combining information from different sources, the influence of noise or changes on individual features can be effectively offset, thereby improving the reliability of predicted pixels. The fused features can more completely describe the content of the current block and / or reference information, thereby effectively improving the accuracy of pixel prediction, reducing residuals, and improving the encoding and decoding quality during video encoding and / or decoding.

[0267] Method D: Determine or generate at least one predicted pixel based on at least one fourth image block feature determined or generated by performing convolution processing on a fusion feature corresponding to at least one reference information of the current block using a neural network model.

[0268] Optionally, a fusion feature corresponding to at least one reference information of the current block can be determined or obtained through fusion processing. The fusion feature is convolved using at least one convolutional layer in the neural network model to determine or generate at least one fourth image block feature; and at least one predicted pixel is determined or generated based on the at least one fourth image block feature. By performing convolution processing on the fusion feature, the neural network model can focus on local features (for example, edges, textures, etc.) and can capture local correlations within the image; the at least one fourth image block feature obtained after the convolution processing can contain rich spatial information and multi-level abstract features, so that the fourth image block feature can more accurately describe the content of the current block and improve the accuracy of pixel prediction.

[0269] Optionally, the fusion feature can be determined or obtained by any one of the above methods A to C.

[0270] Optionally, the neural network model may include at least one convolutional layer. Optionally, the neural network model may be a convolutional neural network.

[0271] Optionally, the neural network model is based on a hybrid convolutional and fully connected layer architecture. This hybrid convolutional and fully connected layer architecture combines the advantages of a convolutional neural network (CNN) and a multilayer perceptron (MLP), comprising both convolutional and fully connected layers. Convolutional layers automatically learn the spatial hierarchical structure of input data, such as features such as edges, texture, and shape in an image, while convolution operations effectively capture local patterns and reduce model complexity through parameter sharing. Fully connected layers flatten the features extracted by the convolutional layers into a one-dimensional vector, which is then fed into a series of fully connected layers that combine low-level features to generate a higher-level abstract representation. This hybrid convolutional and fully connected layer neural network model combines the advantages of both different layer types, efficiently extracting features from input data and making accurate predictions based on these features. Optionally, the hybrid convolutional and fully connected layer neural network model comprises an input layer, at least one convolutional layer, at least one fully connected layer, and an output layer.

[0272] Optionally, referring to FIG12 , a neural network model based on a hybrid convolutional and fully connected layer includes: an input layer, multiple hidden layers, and an output layer, wherein the hidden layers include: a convolutional layer, a fully connected layer (i.e., a linear layer), and an activation function layer (e.g., a LeakyReLU layer). Pixels of a reference region of a current block and pixels of a reference block are input into the neural network model based on a hybrid convolutional and fully connected layer. The reference region includes: a reference region A consisting of pixels above the current block, a reference region B consisting of pixels to the left of the current block, and a reference region C consisting of pixels above the left of the current block. The reference blocks include a first reference block and a second reference block. Convolution processing is performed on the pixels of the first reference block and the pixels of the second reference block through at least one convolutional layer to obtain at least one first image feature. The pixels of the reference region are processed through a hybrid convolutional and fully connected layer method and processed through a ReLU activation function through an activation function layer to obtain at least one second image block feature. The first image block feature and the second image block feature are fused to obtain at least one third image block feature. The at least one third image block feature is processed through at least one convolutional layer to obtain at least one predicted pixel.

[0273] Optionally, referring to FIG13 , a neural network model based on a hybrid convolutional and fully connected layer includes: an input layer, multiple hidden layers, and an output layer. The hidden layers include: a convolutional layer, a fully connected layer (i.e., a linear layer), and an activation function layer (e.g., a LeakyReLU layer). Pixels of a reference region of a current block and pixels of a reference block are input into the neural network model based on a hybrid convolutional and fully connected layer. The reference region includes: a reference region A consisting of pixels above the current block, a reference region B consisting of pixels to the left of the current block, and a reference region C consisting of pixels to the upper left of the current block. The reference blocks include a first reference block and a second reference block. The pixels of the reference block are convolved using at least one convolutional layer to obtain at least one first image block feature. The pixels of the reference region are processed using at least one fully connected layer and combined with an activation function layer to perform activation function processing to obtain at least one second image block feature. The at least one first image block feature and the at least one second image block feature are fused to obtain at least one third image block feature. Convolution processing is performed on at least one third image block feature through at least one convolution layer to restore the feature into an output feature of the same size as the target label to obtain at least one predicted pixel.

[0274] Alternatively, when the input data is the pixels of a reference block and the processing device is an encoder, there may be multiple candidate combinations of reference blocks, and corresponding features need to be extracted for each combination. However, if this is processed through a fully connected layer, the computational efficiency may be low. Convolutional layers, on the other hand, have the advantages of having fewer parameters and being better at capturing local features. They can efficiently extract information about the block to be fused while requiring less computing resources. Therefore, processing the pixels of the reference block through a convolutional layer can improve processing efficiency.

[0275] Optionally, when the input data is pixels in a reference area, and the pixels in the reference area are reconstructed reference pixels, since the reconstructed reference pixels provide information already present within the frame, the extraction efficiency requirement is relatively low. Therefore, a fully connected layer can be used to perform more refined feature extraction and improve the accuracy of pixel prediction. At the same time, the fully connected layer has greater flexibility and can process pixels in reference areas of different sizes (e.g., different numbers of layers), improving the flexibility of pixel prediction and thereby improving the encoding and decoding quality during the video encoding and / or decoding process.

[0276] Alternatively, a neural network model based on a hybrid convolutional and fully connected layer can combine the advantages of two different types of layer structures, which can not only efficiently extract features from the input data, but also make more accurate predictions based on the features.

[0277] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0278] Optionally, the first model is determined or obtained based on the width and height of at least one reference area of ​​the current block. Optionally, when the width and height of the reference area are both less than or equal to X1 (e.g., 8), and / or when the height and width of the output predicted pixel are both less than or equal to X1 (e.g., 8), the first model is a neural network model based on a hybrid convolutional and fully connected layer, and the hidden layer structure of the first model is shown in Table 6 below:

[0279] Table 6

[0280] Optionally, when at least one of the width and height of the reference area is greater than X1 (for example, 8), and / or when at least one of the height and width of the output predicted pixel is greater than X1 (for example, 8), the first model is a neural network model based on a hybrid convolution and fully connected layer, and the hidden layer structure of the first model is shown in Table 7 below:

[0281] Table 7

[0282] In this embodiment, by performing convolution processing on the fused features, the neural network model can focus on local features (for example, edges, textures, etc.), and thus can capture local correlations within the image; at least one fourth image block feature obtained by the convolution processing can contain rich spatial information and multi-level abstract features, so that the fourth image block feature can more accurately describe the content of the current block and improve the accuracy of pixel prediction.

[0283] Optionally, the first model is determined or obtained based on the reference information.

[0284] Optionally, the first model may be determined from a group of neural network models based on the type, content, etc. of the reference information.

[0285] Optionally, the structure and / or type of the first model may be determined based on the type and content of the reference information. For example, when the reference information includes pixels of at least one reference region of the current block, the first model may be determined to include at least one fully connected layer. When the reference information includes pixels of at least one reference block of the current block, the first model may be determined to include at least one convolutional layer.

[0286] Optionally, the first model is determined according to at least one of a width, a height, a size and an area of ​​the reference region and / or the reference block.

[0287] In this embodiment, the first model can be determined or obtained based on the reference information, so that the first model can more accurately capture the characteristics of the input data, improve the prediction effect, and thus improve the encoding and decoding quality during the video encoding and / or decoding process.

[0288] Optionally, first, N data sets are obtained. After obtaining the N data sets, the N data sets are divided into two parts, one for training and the other for testing. Optionally, the number of training data sets is N1, and the number of testing data sets is N2, where N1 + N2 = N. Next, a neural network is trained using the training data sets, and the trained neural network is tested using the testing data sets.

[0289] Optionally, pixel value offset correction is performed on each of the N1 training datasets, and reference pixels of a reference area input to the neural network are obtained based on the position information of each dataset. The obtained reference pixels are input to the neural network to train the neural network. After the neural network training is completed, the trained neural network is tested using each of the N test datasets to determine whether the trained neural network can make accurate predictions and meet the requirements.

[0290] Alternatively, if a matching neural network is trained for different image block sizes, the image block size of each of the N1 training datasets mentioned above is determined. Based on the image block size, the reference pixels of the reference area and / or the pixels of the reference block corresponding to the training dataset are input into the matching neural network. This neural network is then trained. Similarly, during the testing phase, the test dataset corresponding to the image library block size is input into the matching neural network for testing.

[0291] Optionally, multiple video sequences for constructing a dataset are first determined. Then, a data extraction module is provided in the encoder and decoder to extract data from the video sequences. The extracted data includes: a frame index (poc), position information (x, y) of an image block (e.g., a coding unit), size (width, height) of the image block, channel type (compidx), split angle (splitdir), pixel value offset (shiftWeighted) and compensation (offsetWeighted), original image pixel value (org) of the image block, and at least one of pixel values ​​of a reference block of the image block.

[0292] Optionally, data is extracted from the multiple video sequences using an encoder and decoder equipped with a data extraction module. The original data is encoded using four QP values ​​(22, 27, 32, and 37), and corresponding bitstream files are determined or generated. A decoder equipped with the data extraction module is used to decode the bitstream files, and decoded reconstructed data is determined or generated.

[0293] Optionally, a target label for pixel prediction is determined or generated using the original pixel value of the current block to guide the training process of the neural network model, thereby determining or obtaining training data.

[0294] Optionally, load data from a pre-built training dataset. The dataset contains the position, size, channel information, partition angle, offset, original pixel value, and pixel value of the reference block of the current block. Normalize the loaded data. Take the pixels of the reference area of ​​the current block as input and the predicted pixels as output. For example, if the input reference area includes reference area A, reference area B, and reference area C, the input sizes are recorded as [B, C 1 ,H 1 ,W 1 ]、[B,C 2 ,H 2 ,W 2 ] and [B,C 3 ,H 3 ,W 3 ].

[0295] Optionally, in the case where the neural network model includes at least one fully connected layer, since the input data of the fully connected layer needs to be a two-dimensional structure of [B, C], the pixels of the reference area A, the reference area B, and the reference area C can be flattened to determine or obtain the structural features of [B, C×H×W], and the flattened features are spliced ​​in the second dimension (i.e., C dimension) to obtain a structure of size [B, C 1 ×H 1 ×W 1 +C 2 ×H 2 ×W 2 +C 3 ×H 3 ×W 3 ] characteristics.

[0296] Alternatively, in the case where the neural network model is a neural network model based on a hybrid convolutional and fully connected layer, the pixels of the reference area and the pixels of the reference block can be used as input, and the predicted pixels can be used as output. The input size is denoted as the reference area A[B,C 1 ,H 1 ,W 1 ], reference area B[B,C 2 ,H 2 ,W 2 ], reference area C[B,C 3 ,H 3 ,W 3 ], the first reference block [B,C 4 ,H 4 ,W 4 ], the second reference block [B,C 5 ,H 5 ,W 5]. For the features corresponding to the first reference block and the second reference block, the features can be extracted by convolution processing to obtain the corresponding [B, C 6 ,H 6 ,W 6 In order to better fuse the obtained features with the features of the reference area, the features of each reference area can be expanded into the shape of [B, C, H, W] and then fused.

[0297] Optionally, for the size where the width w of the current block is not equal to the height h, both w×h and h×w data in the dataset are included in the training, and the h×w data can be transposed to obtain w×h. For the size where the width w is equal to the height h, in addition to using the original data for training, the data after the original data itself is transposed can be added as training data. Optionally, in the actual encoding and decoding process, since some positions may not have reconstructed pixels, that is, the encoder may not be able to obtain complete reference area information, resulting in missing pixel values. In order to simulate this situation and improve the generalization ability of the model, the data can be truncated and the pixel filling process in the video codec can be simulated to improve data diversity and adapt to situations where complete reference information cannot be obtained.

[0298] Optionally, set the loss function for model training; use a learning rate scheduling strategy to decay the learning rate every preset round to promote model convergence. For example, the loss function L SmoothL1 (x) is:

[0299] Optionally, the input data from the training data is fed into the neural network model, and a forward pass is performed to obtain a prediction. The prediction is compared with the target label to determine or obtain the training loss. The gradient is calculated using a backpropagation algorithm, and the model parameters are updated using an optimizer to minimize the loss function. After a preset number of iterations, the model switches to evaluation mode, and a forward pass is performed on the validation set to calculate the validation loss. By comparing the training loss and validation loss, the model's generalization ability is monitored to prevent overfitting or underfitting. After training is complete, a trained neural network model is obtained.

[0300] Optionally, after the neural network model is trained to achieve optimal performance, the neural network model may be used to perform inter-frame prediction processing during the inter-frame prediction phase of the encoding and decoding process. For example, at least one predicted pixel may be determined or generated based on at least one reference information of the current block, including pixels of at least one reference region of the current block and / or pixels of at least one reference block of the current block, and the neural network model.

[0301] Optionally, to reduce the time complexity associated with neural network model inference, the neural network model can be optimized for video codecs. For example, through model quantization, floating-point parameters and calculations in the neural network model are converted to integer form, reducing the model's storage requirements and accelerating the inference process. For example, by statically quantizing the trained model, the trained floating-point weights and activation function output values ​​are converted to 8-bit integers, ensuring that the model uses only integer operations during inference. This not only significantly improves the model's inference speed but also significantly reduces memory usage, making it particularly suitable for embedded devices and mobile applications.

[0302] Alternatively, a Small Adhoc Deep-Learning Library (SADL) for deep learning that supports efficient integer operations can be selected to ensure that, based on the quantized model, the reasoning process can make full use of hardware acceleration features, such as single instruction multiple data (SIMD) instruction set and multi-threaded parallel computing. Alternatively, unnecessary functional modules and redundant codes in the SADL library can be eliminated by analyzing actual needs, and only the parts directly related to the neural network model reasoning in the embodiment of the present application are retained. This not only reduces the size of the library and reduces memory usage, but also reduces the overhead of function calls, further improving reasoning efficiency. Alternatively, customized optimization can also be performed for the specific structure of the model; for example, the calculation process of the fully connected layer and the convolutional layer is optimized, the number of memory accesses is reduced, and the cache hit rate is improved, thereby speeding up reasoning.

[0303] Optionally, in order to speed up the inference process, a neural network model structure that is as simple as possible can be set. Under the premise of ensuring the prediction accuracy of the model, the computational complexity and parameter quantity of the model can be reduced by reducing the number of network layers, reducing the number of neurons or channels in each layer, and adopting a lightweight network architecture. And / or, pruning technology can be used to remove neurons or channels that have little impact on the final output to further reduce the size of the model. In addition, the parameter sharing and reuse mechanism of the convolutional layer can be used to reduce the number of parameters of the overall model while maintaining the necessary expressive power. Optimizing the choice of activation function and avoiding the use of activation functions with high computational complexity can also effectively accelerate the nonlinear transformation process.

[0304] Optionally, the processing method of the present application can be integrated into the inter-frame prediction module of the encoder and / or decoder; for example, an inter-frame prediction mode based on the first parameter is defined in the inter-frame prediction module as a class in the GPM model, responsible for managing and executing the inference process of the neural network model, including: model initialization, reference data extraction, inference process, predicted pixel output, etc. Optionally, the inter-frame prediction module is a module shared by the encoder and decoder, including: motion search, motion compensation, etc.

[0305] Optionally, when the integrated encoder and / or decoder starts, neural network models suitable for different image block sizes can be loaded. For example, a set of different model sizes can be predefined to accommodate different current block and prediction block size requirements. For each predefined size, a corresponding model file path is constructed, the model file is read from the specified path, and loaded into memory. If the model file cannot be read, the system will log an error and terminate execution. After successful loading, the model is stored in the class's internal model list for subsequent inference.

[0306] Optionally, before performing inter-frame prediction, at least one reference region of the current block is determined or obtained for inference by the neural network model. Optionally, the width and height of the current block can be determined, and the total length of the reference region can be selected accordingly. Next, pixels in the reference region are collected. For example, these may include the upper neighboring pixels, the left neighboring pixels, and the upper left neighboring pixels of the current block. The upper neighboring pixels can be formed by collecting pixel data from the top boundary of the current block from bottom to top and from left to right, forming a matrix; the left neighboring pixels can be formed by collecting pixel data from the left boundary of the current block from right to left and from top to bottom, forming a matrix; and the upper left neighboring pixels can be formed by collecting pixel data from the top boundary of the current block from bottom to top and from right to left, forming a matrix. If valid pixel data is missing in certain locations, it can be filled with adjacent valid pixels. Optionally, the collected pixels in the reference region and the reference block are normalized according to the bit depth to meet the input requirements of the neural network model. The normalized pixels in the reference region and the reference block are packaged into input datasets, respectively, in preparation for the subsequent inference process.

[0307] Optionally, once the input data is ready, a neural network model can be used for inference to generate predicted pixels. Optionally, a corresponding neural network model is selected based on at least one of the current block's width, height, block size, and block area. The input data is fed into the selected model, ensuring that the quantity and size of the input data match the model's expectations. If the input data also includes other supplementary data, such as reference blocks, this can be directly obtained during the use of the GPM tool in the inter-frame prediction module.

[0308] Optionally, input data is passed to the model for initialization, ensuring that the model is in a state ready for inference. The model's inference function is called to process the input data (i.e., pixels in the reference region) and generate output results (i.e., predicted pixels). If any problems occur during inference, an error message is returned. The predicted pixels are extracted from the model's inference results and stored in a pixel buffer for subsequent use.

[0309] Optionally, the pixel buffer is resized to accommodate all generated predicted pixels based on at least one of the width, height, block size, and block area of ​​the current block. The predicted pixels output by inference are assigned one by one to the pixel buffer to ensure that each pixel in the current block has a corresponding pixel value.

[0310] Optionally, in the encoder, the integration of the processing method of the embodiment of the present application is mainly concentrated in the mode discrimination and signaling transmission parts. The encoder's prediction mode discrimination can select whether to adopt the inter-frame prediction mode based on the first parameter of the embodiment of the present application based on the principle of minimizing rate-distortion cost. The mode selected by the encoder can be transmitted through signaling to inform the decoder whether to use the inter-frame prediction mode based on the first parameter of the embodiment of the present application for decoding.

[0311] Optionally, the encoder can generate a geometric partitioning candidate list (i.e., a GPM candidate list) and estimate the rate-distortion cost. For example, a GPM candidate list is created to store the prediction results of different candidate modes. By calling the motion estimation module, the encoder obtains the available GPM candidate modes for the current block, which contain different motion vector combinations. For each candidate mode, the sum of absolute differences of the corresponding prediction errors can be estimated using conventional methods and preliminarily sorted.

[0312] Optionally, in the encoder, it is possible to check whether the current block has an available reference area to determine whether the inter-frame prediction mode based on the first parameter of the embodiment of the present application can be applied. If the reference area is available, the encoder can use a pre-trained neural network model to predict the pixels of the current block. For example, the encoder inputs the pixels in the reference area consisting of the upper adjacent pixels, left adjacent pixels and upper left adjacent pixels of the current block and the pixels of the reference block into the neural network to generate predicted pixels. Based on the predicted pixels, a prediction block is generated. The error between the predicted block and the original image (i.e., the current block) is calculated. The prediction error is combined with its bit rate overhead to calculate its overall rate-distortion cost.

[0313] Optionally, in the encoder, all candidate GPM modes can be sorted according to their estimated rate-distortion costs, from low to high. Optionally, an early stopping strategy can be applied, that is, if the cost of certain candidate modes is significantly higher than the current best mode, further evaluation of these modes is skipped to improve coding efficiency. The encoder can select the geometric partitioning mode with the lowest rate-distortion cost as the final coding mode. If the inter-frame prediction mode based on the first parameter of an embodiment of the present application shows a lower cost, this mode is preferably selected.

[0314] Optionally, the encoder can determine or obtain the residual between the predicted block and the current block, and transform, quantize, and entropy encode the residual to achieve effective data compression. The encoder transmits the selected mode information through the code stream; for example, if the mode corresponding to the processing method of the embodiment of the present application is selected, the signaling corresponding to the inter-frame prediction mode based on the first parameter of the embodiment of the present application and the motion vector combination candidate index are transmitted, without transmitting the original GPM division method index, thereby saving the signaling required to transmit the division method.

[0315] Optionally, in the decoder, the integration of the processing method of the embodiment of the present application is mainly focused on the reading of signaling and the application of the mode. The decoder can first parse the signaling information from the code stream, and decide which motion compensation mode to adopt based on the parsed mode flag. If the flag corresponding to the inter-frame prediction mode based on the first parameter of the present application is set, the decoder calls the code of the inter-frame prediction mode based on the first parameter of the present application integrated into the inter-frame prediction module, and uses the predicted pixels predicted by the neural network model to reconstruct the final image block (i.e., the prediction block).

[0316] Optionally, the inter-frame prediction performance of the processing method of the application in video encoding and decoding can be evaluated based on the Common Test Conditions (CTC). The test process strictly follows the test sequence, quantization parameter QP, encoding profile and evaluation indicators defined in CTC to ensure the reliability and repeatability of the experimental results. For example, the test process includes: Step 1, encoding configuration. According to the definition in CTC, select a test sequence suitable for random access (RA) and low delay (LD) conditions. For each test sequence, under each test condition, the encoding profile defined in CTC is used for encoding to ensure the consistency of the encoding process. Step 2, use an encoder that integrates the processing method of this application to encode all test sequences using four QP values ​​(22, 27, 32, 37) to generate corresponding bitstream files. Use the same version of the decoder to decode the generated bitstream file and restore the reconstructed video sequence. Step 3, performance evaluation. Extract the test results based on the encoding and decoding logs, including bit rate, peak signal-to-noise ratio and encoding and decoding time. The rate-distortion performance is calculated based on the bit rate and peak signal-to-noise ratio results at different QPs to evaluate the bit rate savings achieved by the proposed method compared to the baseline encoder and / or decoder.

[0317] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0318] Method eight, regression model;

[0319] Optionally, at least one reference information of the current block (for example, the at least one reference information of the current block includes pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block of the current block) can be used as input to the regression model, and the relationship between the reference information of the current block and the predicted pixels learned by the regression model is used to output at least one predicted pixel.

[0320] Alternatively, a regression model is a statistical method used to establish the relationship between independent variables (features) and dependent variables (targets). A regression model can construct a mathematical function by analyzing the relationship between known data points, so that the function can predict the value of new unknown data points as accurately as possible.

[0321] Optionally, the regression model includes at least one of a linear regression model, a polynomial regression model, a logistic regression model, a support regression model, a decision tree regression model, a random forest regression model, and a gradient boosting regression model.

[0322] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0323] In this embodiment, a regression model is used to directly predict pixels in the current block using at least one reference information of the current block. This avoids the impact of conventional linear partitioning on prediction accuracy while taking into account the relationship between the current block and its reference region and reference blocks, thereby improving the prediction performance of inter-frame prediction. Furthermore, this effectively reduces the signaling required for partitioning, lowering signaling overhead and improving video compression efficiency. Furthermore, the regression model has low computational complexity, enabling rapid completion of the training and prediction processes.

[0324] Method nine, based on the matrix mathematical model;

[0325] Optionally, the matrix-based mathematical model may be a mathematical model constructed by obtaining a matrix by simplifying a neural network model.

[0326] Optionally, at least one reference information of the current block (for example, the at least one reference information of the current block includes pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block of the current block) can be used as input of a matrix-based mathematical model, and the relationship between the reference information of the current block and the predicted pixels learned by the matrix-based mathematical model is used to output at least one predicted pixel.

[0327] Alternatively, a mathematical model is a type of model described in mathematical language. A mathematical model can be one or a set of algebraic equations, differential equations, difference equations, integral equations, or statistical equations, or some appropriate combination thereof, which quantitatively or qualitatively describe the interrelationships or causal relationships between system variables. In addition to mathematical models described using equations, there are also models described using other mathematical tools, such as algebra, geometry, topology, and mathematical logic. It is understood that mathematical models describe the behavior and characteristics of a system rather than its actual structure.

[0328] Optionally, a matrix-based mathematical model refers to a type of model that uses matrix operations and linear algebra principles to describe and solve problems. It represents data as matrices and uses matrix operations (such as multiplication, inverse, eigenvalue decomposition, etc.) to perform calculations and analysis. It can efficiently process large-scale data sets and complex mathematical relationships.

[0329] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0330] In this embodiment, a matrix-based mathematical model is used to directly predict the weights of the current block using pixels in the reference region of the current block. This avoids the impact of conventional linear partitioning on prediction accuracy while taking into account the relationship between the current block, its reference region, and the reference block, thereby improving the prediction effect of inter-frame prediction. Furthermore, this can effectively reduce the signaling required for partitioning, lower signaling overhead, and improve video compression efficiency. Furthermore, the matrix-based mathematical model can effectively represent multidimensional data, reducing complexity and improving interpretability. It can also quickly execute large amounts of computational tasks, thereby improving the quality of video encoding and decoding.

[0331] Method 10: mathematical model based on lookup table;

[0332] Optionally, at least one reference information of the current block (for example, the at least one reference information of the current block includes pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block of the current block) can be used as input to a mathematical model based on a lookup table, and the relationship between the reference information of the current block and the predicted pixels learned by the mathematical model based on the lookup table is used to output at least one predicted pixel.

[0333] Alternatively, a lookup table-based mathematical model is a technique that enables fast querying by precalculating and storing a series of input-output mappings. A lookup table is essentially a two-dimensional or multi-dimensional array, where each element corresponds to an output value for a specific input condition. In practical applications, when the output corresponding to a specific input needs to be calculated, the result can be directly read from the lookup table without having to perform complex calculations again.

[0334] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0335] In this embodiment, a lookup table-based mathematical model utilizes pixels in the reference region of the current block to directly predict the weight of the current block. This avoids the impact of conventional linear partitioning on prediction accuracy while also taking into account the relationship between the current block, its reference region, and the reference block, thereby improving the prediction performance of inter-frame prediction. Furthermore, this can effectively reduce the signaling required for partitioning, lower signaling overhead, and improve video compression efficiency. Furthermore, by completing most of the computational work in advance and storing the results in the lookup table, the lookup table-based mathematical model can significantly reduce the amount of computation at runtime, lowering the processor burden.

[0336] Method 11: mathematical model based on normalization operation.

[0337] Optionally, at least one reference information of the current block (for example, the at least one reference information of the current block includes pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block of the current block) can be used as input of a mathematical model based on a normalization operation, and the relationship between the reference information of the current block and the predicted pixels learned by the mathematical model based on the normalization operation is used to output at least one predicted pixel.

[0338] Alternatively, mathematical models based on normalization operations refer to methods that simplify calculations, improve model performance, and enhance the interpretability of results by converting input data into a standardized range. Normalization operations are often used to process data at different scales to ensure that each feature is compared or combined on the same basis.

[0339] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0340] In this embodiment, by using a mathematical model based on normalization operations and directly performing weighted prediction on the current block using pixels in the reference region of the current block, the effects of conventional linear partitioning on prediction accuracy can be avoided while also taking into account the correlation between the current block, its reference region, and the reference block, thereby improving the prediction performance of inter-frame prediction. Furthermore, this can effectively reduce the signaling required for partitioning, lower signaling overhead, and improve video compression efficiency.

[0341] Fourth embodiment

[0342] Based on any of the above embodiments, a fourth embodiment is proposed.

[0343] In this embodiment, the processing method further includes at least one of the following methods 12 to 14:

[0344] Method 12, at least one predicted pixel is a first predicted pixel;

[0345] Optionally, at least one first predicted pixel is determined or generated according to the first parameter.

[0346] Optionally, at least one first predicted pixel is determined or generated according to at least one reference information of the current block and a first model.

[0347] Optionally, at least one first predicted pixel is determined or generated based on the first model and pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block of the current block.

[0348] In this embodiment, the reference region is determined or obtained according to any one of the first to sixth methods in the second embodiment. The first model is determined or obtained according to any one of the seventh to eleventh methods in the third embodiment.

[0349] Method 13, determining or generating a first prediction mode according to at least one first prediction pixel;

[0350] Optionally, in order to further improve pixel prediction accuracy, a first prediction mode may be further determined or generated based on at least one first pixel, and further prediction processing may be performed using the first prediction mode.

[0351] Optionally, the first prediction mode corresponding to the first predicted pixel may be obtained according to a preset mapping / correspondence rule.

[0352] In a fourteenth method, prediction processing is performed on the current block according to the first prediction mode to obtain at least one second predicted pixel.

[0353] Optionally, prediction processing is performed on the current block according to the first prediction mode to obtain at least one second predicted pixel, and a prediction block of the current block is determined or generated according to the at least one second predicted pixel.

[0354] Optionally, a prediction block of the current block may be determined or generated based on the predicted pixels, and a residual block between the prediction block and the current block may be calculated, which is then transformed and quantized and encoded by an entropy encoder to form an encoded bit stream.

[0355] Alternatively, in the first prediction mode, the current block may be processed using other models. For example, a more complex interpolation filter (e.g., a high-order polynomial filter, an adaptive interpolation filter, etc.) may be used to process the current block to determine or generate at least one second predicted pixel to improve the matching degree between the predicted pixel and the pixel of the original image block (i.e., the current block).

[0356] Optionally, the first prediction mode can be an angle prediction mode, and the predicted pixel is further predicted by the angle prediction mode; by taking into account the directional characteristics in the image content (such as horizontal, vertical, diagonal, etc.), the direction that best reflects these characteristics is selected for prediction to obtain at least one second predicted pixel.

[0357] Optionally, at least one first predicted pixel is determined or generated based on the first model and at least one reference information of the current block, including pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block of the current block; a first prediction mode is determined or generated based on the at least one first predicted pixel; and according to the first prediction mode, prediction processing is performed on the current block to obtain at least one second predicted pixel.

[0358] In this embodiment, the reference region is determined or obtained according to any one of the first to sixth methods in the second embodiment. The first model is determined or obtained according to any one of the seventh to eleventh methods in the third embodiment.

[0359] In this embodiment, a first prediction mode can be further determined or generated based on the first predicted pixel, and further prediction processing can be performed on the current block based on the first prediction mode to obtain a second predicted pixel with higher accuracy. By performing multiple prediction processes on the current block, the accuracy of pixel prediction can be gradually improved, thereby improving the encoding and decoding quality during the video encoding and / or decoding process.

[0360] Optionally, the processing method further includes at least one of the following methods 15 to 17:

[0361] Mode 15: determining a residual corresponding to at least one second predicted pixel according to a syntax element obtained from a bitstream;

[0362] Optionally, when the processing device is a decoder on the decoding side, the decoder may decode the encoded code stream to obtain syntax elements; and determine the residual corresponding to the at least one second predicted pixel according to the syntax elements.

[0363] Optionally, syntax elements refer to various information units generated and embedded in the bitstream during the encoding process. These units are used to guide the decoder to correctly reconstruct the original video content. Each syntax element usually corresponds to a specific function or parameter setting, ensuring consistency between the encoder and decoder.

[0364] Optionally, the residual refers to the difference between the original image block and its predicted value (e.g., predicted block, predicted pixel). Specifically, during the video encoding process, in order to reduce redundancy and improve compression efficiency, the encoder calculates the difference between the actual pixel value and the predicted value. This difference is called the "residual."

[0365] In this embodiment, when the processing device is a decoder on the decoding side, the decoder can decode the encoded code stream to obtain syntax elements and quickly determine the residual corresponding to the second predicted pixel for decoding, thereby improving decoding efficiency.

[0366] Mode 16, determining or generating a reconstructed pixel based on the residual and at least one second predicted pixel;

[0367] Optionally, when the processing device is a decoder on the decoding side, the decoder may add the residual to the second predicted pixel, thereby determining or generating the reconstructed pixel.

[0368] Optionally, the reconstructed pixel refers to a pixel value that is finally generated and as close as possible to the original image, which can be obtained by adding the decoded residual to the predicted pixel.

[0369] Optionally, at least one first predicted pixel is determined or generated based on the first model and at least one reference information of the current block, including pixels of at least one reference region of the current block and / or pixels of at least one reference block of the current block; a first prediction mode is determined or generated based on the at least one first predicted pixel; prediction processing is performed on the current block based on the first prediction mode to obtain at least one second predicted pixel; a residual corresponding to the at least one second predicted pixel is determined based on syntax elements obtained from the bitstream; and a reconstructed pixel is determined or generated based on the residual and the at least one second predicted pixel.

[0370] In this embodiment, the reference region is determined or obtained according to any one of the first to sixth methods in the second embodiment. The first model is determined or obtained according to any one of the seventh to eleventh methods in the third embodiment.

[0371] In this embodiment, when the processing device is a decoder on the decoding side, the decoder determines or generates a reconstructed pixel based on the residual and at least one second predicted pixel to achieve fast decoding and improve decoding efficiency.

[0372] In a seventeenth method, a corresponding residual is determined or generated according to at least one original pixel and at least one second predicted pixel corresponding to the current block.

[0373] Optionally, when the processing device is an encoder on the encoding side, the encoder determines or generates a residual corresponding to the second predicted pixel according to at least one original pixel and at least one second predicted pixel corresponding to the current block.

[0374] Optionally, the encoder uses a difference between at least one original pixel corresponding to the current block and at least one second predicted pixel as a residual corresponding to the second predicted pixel.

[0375] Optionally, at least one first predicted pixel is determined or generated based on the first model and at least one reference information of the current block, including pixels of at least one reference region of the current block and / or pixels of at least one reference block of the current block; a first prediction mode is determined or generated based on the at least one first predicted pixel; prediction processing is performed on the current block based on the first prediction mode to obtain at least one second predicted pixel; and a corresponding residual is determined or generated based on the at least one original pixel corresponding to the current block and the at least one second predicted pixel.

[0376] In this embodiment, the reference region is determined or obtained according to any one of the first to sixth methods in the second embodiment. The first model is determined or obtained according to any one of the seventh to eleventh methods in the third embodiment.

[0377] In this embodiment, when the processing device is an encoder on the encoding side, the encoder can determine or generate a corresponding residual based on at least one original pixel and at least one second predicted pixel corresponding to the current block; since the accuracy of the second predicted pixel is higher, the residual corresponding to the second predicted pixel is smaller, and the amount of data required to represent the residual in the code stream is smaller, thereby reducing the overall bit rate and improving the efficiency of video encoding and / or decoding.

[0378] Optionally, the first model is determined based on at least one of the width, height, size, and area of ​​at least one reference region of the current block. Optionally, a first model corresponding to the target model type is determined from a plurality of model types based on at least one of the width, height, size, and area of ​​at least one reference region of the current block.

[0379] Optionally, based on at least one of the geometric properties of at least one reference area of ​​the current block (i.e., width, height, block size and block area), a model type that is most suitable for processing these features is selected, so that the first model can more accurately reflect the characteristics of the input data and improve the prediction effect.

[0380] Optionally, the structure of the first model is determined according to at least one of width, height, size and area of ​​at least one reference region of the current block.

[0381] Optionally, the specific structure of the selected model is adjusted according to at least one of the geometric properties of at least one reference area of ​​the current block (i.e., width, height, block size, and block area). For example, when the first model is a neural network model, the number of hidden layers (e.g., convolutional layers, fully connected layers), filter size, pooling strategy, etc. can be adjusted according to at least one of the width, height, size, and area of ​​at least one reference area. For example, when the first model is a mathematical model based on a lookup table, the corresponding weight mapping can be pre-calculated and stored according to at least one of the width, height, size, and area of ​​at least one reference area. For example, when the first model is a mathematical model based on a matrix, a specific transformation matrix can be designed to adapt to the above-mentioned geometric properties according to at least one of the width, height, size, and area of ​​at least one reference area.

[0382] Optionally, if the width, height, size, or area of ​​the reference region of the current block does not match any of the multiple target model types, the reference region corresponding to the current block is transposed or downsampled to convert it to a reference region with a width, height, size, or area that matches at least one of the multiple target model types. In this way, more reference regions can be matched to target models or target model types through transposition or downsampling, thereby reducing the number of target models or target model types overall.

[0383] Optionally, the first model is determined based on at least one of the width, height, block size, and block area of ​​the current block. Optionally, at least one of the width, height, block size, and block area of ​​the current block is associated with at least one reference region of the current block; for example, the at least one reference region can be determined or obtained based on at least one of the width, height, block size, and block area of ​​the current block. Therefore, the first model can be determined or obtained based on at least one of the width, height, block size, and block area of ​​the current block.

[0384] Optionally, a first model corresponding to the target model type is determined from a plurality of model types according to at least one of a width, a height, a block size, and a block area of ​​the current block.

[0385] Optionally, based on at least one of the geometric properties of the current block (i.e., width, height, block size, and block area), a model type that is most suitable for processing these features is selected, so that the first model can more accurately reflect the characteristics of the input data and improve the prediction effect.

[0386] Optionally, the structure of the first model is determined according to at least one of the width, height, block size and block area of ​​the current block.

[0387] Optionally, the specific structure of the selected model is adjusted according to at least one of the geometric properties of the current block (i.e., width, height, block size, and block area). For example, when the first model is a neural network model, the number of hidden layers (e.g., convolutional layers, fully connected layers), filter size, pooling strategy, etc. can be adjusted according to at least one of the width, height, block size, and block area. For example, when the first model is a mathematical model based on a lookup table, the corresponding weight mapping can be pre-calculated and stored according to at least one of the width, height, block size, and block area. For example, when the first model is a mathematical model based on a matrix, a specific transformation matrix can be designed to adapt to the above-mentioned geometric properties according to at least one of the width, height, block size, and block area.

[0388] Optionally, the structure of the first model is determined according to at least one of width, height, size and area of ​​at least one reference block of the current block.

[0389] In this embodiment, based on at least one of the width, height, size and area of ​​the current block, at least one reference area of ​​the current block, and at least one reference block of the current block, a model that is most suitable for processing these features is determined or obtained, so that the first model can more accurately reflect the characteristics of the input data, improve the prediction effect, and thereby improve the encoding and decoding quality in the video encoding and / or decoding process.

[0390] Fifth embodiment

[0391] The present application also provides a processing device, referring to FIG14 , which includes:

[0392] The processing module A10 is configured to determine or generate at least one predicted pixel according to a first parameter.

[0393] Optionally, the first parameter includes at least one reference information of the current block and a first model.

[0394] Optionally, the at least one reference information of the current block includes pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block of the current block.

[0395] Optionally, the first model includes at least one of the following:

[0396] Neural network models;

[0397] Regression models;

[0398] Matrix-based mathematical models;

[0399] Mathematical models based on lookup tables;

[0400] Mathematical model based on normalization operation.

[0401] Optionally, the processing module A10 is further configured to:

[0402] Determine or generate at least one reference area according to at least one of an upper adjacent pixel, an upper non-adjacent pixel, a left adjacent pixel, a left non-adjacent pixel, an upper left adjacent pixel, and an upper left non-adjacent pixel of the current block;

[0403] Determine or generate at least one reference area according to at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block, and a default block corresponding to the current block;

[0404] Determine or generate at least one reference area according to at least one of the width, height, block size and block area of ​​the current block;

[0405] Determine or generate at least one reference area based on a candidate block determined or generated based on a candidate motion vector or a candidate block vector of a current block;

[0406] If the first information of the current block satisfies the first condition, the reference area is the first reference area;

[0407] If the first information of the current block does not meet the first condition, the reference area is the second reference area.

[0408] Optionally, the processing module A10 is further configured to:

[0409] Determine or generate at least one predicted pixel based on at least one first image block feature determined or generated by convolving at least one reference information of the current block using at least one convolutional layer of a neural network model;

[0410] Determine or generate at least one predicted pixel based on at least one second image block feature determined or generated by performing full-connection processing on at least one reference information of the current block using at least one fully-connected layer of a neural network model;

[0411] Determine or generate at least one predicted pixel based on a fusion feature determined or generated by fusing at least one third image block feature corresponding to at least one reference information of the current block using a neural network model;

[0412] At least one predicted pixel is determined or generated based on at least one fourth image block feature determined or generated by performing convolution processing on a fused feature corresponding to at least one reference information of the current block using a neural network model.

[0413] Optionally, the first parameter includes a first prediction mode, and the processing module A10 is further configured to:

[0414] The current block is processed according to the first prediction mode to determine or generate at least one predicted pixel.

[0415] Optionally, the at least one predicted pixel is a first predicted pixel.

[0416] Optionally, the processing module A10 is further configured to:

[0417] A first prediction mode is determined or generated according to at least one first prediction pixel.

[0418] Optionally, the processing module A10 is further configured to:

[0419] According to the first prediction mode, prediction processing is performed on the current block to obtain at least one second predicted pixel.

[0420] Optionally, the processing module A10 is further configured to:

[0421] A residual corresponding to at least one second predicted pixel is determined according to a syntax element obtained from the bitstream.

[0422] Optionally, the processing module A10 is further configured to:

[0423] A reconstructed pixel is determined or generated according to the residual and at least one second predicted pixel.

[0424] Optionally, the processing module A10 is further configured to:

[0425] A corresponding residual is determined or generated according to at least one original pixel and at least one second predicted pixel corresponding to the current block.

[0426] The processing device provided in the embodiment of the present application has similar implementation principles and beneficial effects to the technical solutions shown in the above-mentioned corresponding method embodiments, and will not be described in detail here.

[0427] An embodiment of the present application further provides a processing device, including a memory and a processor, wherein a processing program is stored in the memory, and when the processing program is executed by the processor, the steps of the processing method in any of the above embodiments are implemented.

[0428] An embodiment of the present application further provides a storage medium having a processing program stored thereon. When the processing program is executed by a processor, the steps of the processing method in any of the above embodiments are implemented.

[0429] In the embodiments of the processing device and storage medium provided in this application, all technical features of any of the above-mentioned processing method embodiments may be included. The expanded and explained contents of the specification are basically the same as those of the embodiments of the above-mentioned methods and will not be repeated here.

[0430] An embodiment of the present application further provides a computer program product, which includes computer program code. When the computer program code runs on a computer, the computer executes the methods in the various possible implementation modes described above.

[0431] An embodiment of the present application also provides a chip, including a memory and a processor, wherein the memory is used to store computer programs, and the processor is used to call and run the computer programs from the memory, so that a device equipped with the chip executes the methods in the various possible implementation modes as described above.

[0432] It is understood that the above scenarios are merely examples and do not limit the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, those skilled in the art will appreciate that with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application will also be applicable to similar technical problems.

[0433] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0434] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0435] The units in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0436] In this application, the same or similar terminology, technical solutions and / or application scenario descriptions are generally only described in detail the first time they appear. When they appear again later, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of this application, for the same or similar terminology, technical solutions and / or application scenario descriptions that are not described in detail later, you can refer to the previous relevant detailed descriptions.

[0437] In this application, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0438] The various technical features of the technical solution of this application can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0439] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as mentioned above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the method of each embodiment of the present application.

[0440] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a storage disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state storage disk Solid State Disk (SSD)).

[0441] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A processing method, characterized in that: Includes steps: S10, determining or generating at least one predicted pixel according to a first parameter.

2. The method according to claim 1, characterized in that The first parameter includes at least one reference information of the current block and a first model.

3. The method according to claim 2, characterized in that The at least one reference information of the current block includes pixels of at least one reference area of ​​the current block and / or pixels of at least one reference block of the current block; and / or the first model includes at least one of the following: Neural network models; Regression models; Matrix-based mathematical models; Mathematical models based on lookup tables; Mathematical model based on normalization operation.

4. The method according to claim 3, characterized in that The method of determining or generating at least one reference area includes at least one of the following: Determine or generate at least one reference area according to at least one of an upper adjacent pixel, an upper non-adjacent pixel, a left adjacent pixel, a left non-adjacent pixel, an upper left adjacent pixel, and an upper left non-adjacent pixel of the current block; Determine or generate at least one reference area according to at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block, and a default block corresponding to the current block; Determine or generate at least one reference area according to at least one of the width, height, block size and block area of ​​the current block; Determine or generate at least one reference area according to the candidate block determined or generated by the candidate motion vector or the candidate block vector of the current block; If the first information of the current block satisfies the first condition, the reference area is the first reference area; If the first information of the current block does not satisfy the first condition, the reference area is the second reference area.

5. The method according to claim 3, characterized in that Step S10 includes at least one of the following: Determine or generate at least one predicted pixel according to at least one first image block feature determined or generated by performing convolution processing on at least one reference information of the current block using at least one convolution layer of the neural network model; Determine or generate at least one predicted pixel according to at least one second image block feature determined or generated by performing full-connection processing on at least one reference information of the current block using at least one fully-connected layer of the neural network model; Determine or generate at least one predicted pixel according to a fusion feature determined or generated by fusing at least one third image block feature corresponding to at least one reference information of the current block using a neural network model; At least one predicted pixel is determined or generated based on at least one fourth image block feature determined or generated by performing convolution processing on a fusion feature corresponding to at least one reference information of the current block using a neural network model.

6. The method according to claim 1, characterized in that The first parameter includes a first prediction mode. Step S10 includes: The current block is processed according to the first prediction mode to determine or generate at least one predicted pixel.

7. The method according to claim 1, characterized in that Also includes at least one of the following: At least one predicted pixel is a first predicted pixel; Determining or generating a first prediction mode according to at least one first prediction pixel; According to the first prediction mode, prediction processing is performed on the current block to obtain at least one second predicted pixel.

8. The method according to claim 7, characterized in that Also includes at least one of the following: Determine a residual corresponding to at least one second predicted pixel according to a syntax element obtained from a bitstream; Determine or generate a reconstructed pixel according to the residual and at least one second predicted pixel; A corresponding residual is determined or generated according to at least one original pixel and at least one second predicted pixel corresponding to the current block.

9. A processing device, characterized in that: include: A memory and a processor, wherein a processing program is stored in the memory, and when the processing program is executed by the processor, the steps of the processing method according to any one of claims 1 to 8 are implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the processing method according to any one of claims 1 to 8.

Images