Processing method, processing equipment and storage medium

Abstract

The invention provides a processing method, processing equipment and a storage medium, and the processing method can be applied to the processing equipment, and comprises the steps: carrying out the prediction processing of a current block according to at least one piece of weight information, and determining or generating a prediction block. According to the technical scheme, the flexibility of video image prediction can be improved, and then the prediction effect of video coding and / or decoding is improved.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a processing method, a processing device and a storage medium. Background Art

[0002] The existing video coding standard (H.266 / VVC) proposes a video frame coding technology. For example, when encoding and decoding a video frame, the protocol divides each frame into different blocks and performs prediction processing and encoding and decoding processing.

[0003] In the process of conceiving and implementing the present application, the inventors found that there are at least the following problems: in the intra-frame prediction and / or inter-frame prediction process, the prediction processing method is relatively simple and not very flexible, which results in poor prediction effect of video coding technology.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the invention

[0005] In response to the above technical problems, the present application provides a processing method, a processing device and a storage medium, which can improve the flexibility of video image prediction and thereby improve the prediction effect of video encoding and / or decoding.

[0006] The present application provides a processing method, which can be applied to a processing device, comprising the steps of:

[0007] S10, performing prediction processing on the current block according to at least one weight information to determine or generate a prediction block.

[0008] Optionally, the weight information is determined or obtained from at least one reference area of ​​the current block.

[0009] Optionally, the reference area is determined or obtained by at least one of the following:

[0010] Determine or obtain at least one reference area according to at least one of an upper adjacent pixel, an upper non-adjacent pixel, a left adjacent pixel, a left non-adjacent pixel, an upper left adjacent pixel, and an upper left non-adjacent pixel of the current block;

[0011] Determine or obtain at least one reference area according to at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block, and a default block corresponding to the current block;

[0012] Determine or obtain at least one reference area according to at least one of the width, height, block size and block area of ​​the current block;

[0013] Determine or obtain at least one reference area according to the candidate block determined or obtained based on the candidate motion vector or the candidate block vector of the current block;

[0014] If the first information of the current block satisfies the first condition, the reference area is the first reference area;

[0015] If the first information of the current block does not satisfy the first condition, the reference area is the second reference area.

[0016] Optionally, the first information of the current block satisfies a first condition, including at least one of the following:

[0017] The value of the first information is a first value;

[0018] The value of the first information is within a first numerical range;

[0019] The value of the first information is greater than or equal to the first threshold;

[0020] The value of the first information is less than or equal to the second threshold.

[0021] Optionally, step S10 includes:

[0022] Determine or obtain at least one weight information according to pixels of at least one reference area of ​​the current block and the first model;

[0023] A prediction block of the current block is determined or generated according to the at least one weight information and at least one reference block of the current block.

[0024] Optionally, the at least one weight information is determined or obtained according to at least one of the following:

[0025] Pixels and a neural network model of at least one reference region of the current block;

[0026] pixels and a regression model of at least one reference region of the current block;

[0027] pixels of at least one reference region of the current block and a matrix-based mathematical model;

[0028] Pixels of at least one reference area of ​​the current block and a mathematical model based on a lookup table;

[0029] Pixels of at least one reference area of ​​the current block and a mathematical model based on a normalization operation.

[0030] Optionally, determining or obtaining at least one weight information according to pixels of at least one reference area of ​​the current block and a neural network model includes:

[0031] Determine or obtain a first neural network model from a set of neural network models;

[0032] At least one weight information is determined or obtained based on pixels of at least one reference area of ​​the current block and the first neural network model.

[0033] Optionally, at least one reference block of the current block is determined or obtained according to an index obtained from a bitstream.

[0034] Optionally, the first model is determined or obtained according to at least one of the following:

[0035] at least one of the width, height, size and area of ​​at least one reference region of the current block;

[0036] At least one of the width, height, block size, and block area of ​​the current block.

[0037] The present application also provides a processing device, including: a memory and a processor, wherein a processing program is stored in the memory, and when the processing program is executed by the processor, the steps of any of the above processing methods are implemented.

[0038] The present application also provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the processing methods described above are implemented.

[0039] As described above, the processing method of the present application can be applied to a processing device, including: performing prediction processing on the current block according to at least one weight information, and determining or generating a prediction block. Through the technical solution of the present application, the prediction processing of the current block according to at least one weight information can be realized, thereby enriching the prediction processing method; and / or, the prediction result can be flexibly adjusted through the weight information, thereby improving the flexibility of video image prediction, and thus improving the prediction effect of video encoding and / or decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0041] Figure 1 A schematic diagram of the hardware structure of a processing device for implementing each embodiment of the present application;

[0042] Figure 2 A communication network system architecture diagram provided for an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the hardware structure of a controller 140 provided in this application;

[0044] Figure 4 A schematic diagram of the hardware structure of a network node 150 provided in this application;

[0045] Figure 5 is a flowchart of a processing method according to the first embodiment;

[0046] Figure 6 is a schematic diagram of an encoding process of an encoder in an image processing method according to the first embodiment;

[0047] Figure 7 is a schematic diagram of a decoding process of a decoder in an image processing method according to the first embodiment;

[0048] Figure 8 is a schematic diagram of a weight matrix according to the first embodiment. Figure 1 ;

[0049] Fig. 9 is a schematic diagram of a weight matrix according to the first embodiment. Figure 2 ;

[0050] Fig.10 is a schematic diagram of a weight matrix according to the first embodiment. Figure 3 ;

[0051] Fig.11 is a schematic diagram of a weight matrix according to the first embodiment. Figure 4 ;

[0052] Fig.12 is a schematic diagram of a weight matrix according to the first embodiment. Figure 5 ;

[0053] Fig.13 is a schematic diagram of a weight matrix according to the first embodiment. Figure 6 ;

[0054] Fig.14 is a schematic diagram of a weight matrix according to the first embodiment. Figure 7 ;

[0055] Fig.15 is a schematic diagram of a weight matrix according to the first embodiment. Figure 8 ;

[0056] Fig.16 is a schematic diagram of a reference area according to the second embodiment Figure 1 ;

[0057] Fig.17 is a schematic diagram of a reference area according to the second embodiment Figure 2 ;

[0058] Fig.18 is a schematic diagram of a reference area according to the second embodiment Figure 3 ;

[0059] Fig.19is a schematic structural diagram of a neural network model based on a fully connected layer according to the fourth embodiment;

[0060] Fig. 20 is a schematic structural diagram of a convolutional layer-based neural network model according to the fourth embodiment;

[0061] Fig.21 is a schematic diagram of the structure of a neural network model based on a hybrid convolutional and fully connected layer according to the fourth embodiment;

[0062] Fig. 22 It is a schematic diagram of a processing module of a processing device.

[0063] The realization of the purpose, functional features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. The above-mentioned drawings have shown clear embodiments of this application, which will be described in more detail later. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0065] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0066] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in this article, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this article, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "at the time of..." or "when..." or "in response to determination". Furthermore, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising", "including" indicate that there are described features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or", "and / or", "including at least one of the following" etc. used in this application can be interpreted as inclusive, or mean any one or any combination. For example, “comprising at least one of the following: A, B, C” means “any of the following: A; B; C; A and B; A and C; B and C; A and B and C”, and for another example, “A, B or C” or “A, B and / or C” means “any of the following: A; B; C; A and B; A and C; B and C; A and B and C”. An exception to this definition will only occur when a combination of elements, functions, steps or operations are inherently mutually exclusive in some manner.

[0067] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0068] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0069] It should be noted that, in this article, step codes such as S10 are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence.

[0070] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0071] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0072] The processing device may be implemented in various forms. For example, the processing device described in this application may include processing devices such as mobile phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, and fixed terminals such as digital TVs and desktop computers.

[0073] The following description will be made by taking a mobile terminal as an example, and those skilled in the art will understand that, in addition to components specifically used for mobile purposes, the construction according to the embodiments of the present application can also be applied to fixed-type terminals.

[0074] See also Figure 1 , which is a schematic diagram of the hardware structure of a mobile terminal for implementing various embodiments of the present application, the mobile terminal 100 may include: RF (Radio Frequency, radio frequency) unit 101, WiFi module 102, audio output unit 103, A / V (audio / video) input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, processor 110, and power supply 111 and other components. Those skilled in the art can understand that Figure 1 The structure of the mobile terminal shown in the figure does not constitute a limitation on the mobile terminal. The mobile terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0075] Combine the following Figure 1 The following is a detailed introduction to the various components of the mobile terminal:

[0076] The radio frequency unit 101 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving the downlink information of the base station, it is sent to the processor 110 for processing; in addition, the uplink data is sent to the base station. Generally, the radio frequency unit 101 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 101 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution), TDD-LTE (Time Division Duplexing-Long Term Evolution), 5G and 6G, etc.

[0077] WiFi is a short-range wireless transmission technology. Mobile terminals can help users send and receive emails, browse web pages, and access streaming media through WiFi module 102. It provides users with wireless broadband Internet access. Figure 1 The WiFi module 102 is shown, but it is understandable that it is not an essential component of the mobile terminal and can be omitted as needed without changing the essence of the invention.

[0078] The audio output unit 103 can convert the audio data received by the RF unit 101 or the WiFi module 102 or stored in the memory 109 into an audio signal and output it as sound when the mobile terminal 100 is in a call signal reception mode, a talk mode, a recording mode, a voice recognition mode, a broadcast reception mode, etc. Moreover, the audio output unit 103 can also provide audio output related to a specific function performed by the mobile terminal 100 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 103 may include a speaker, a buzzer, etc.

[0079] The A / V input unit 104 is used to receive audio or video signals. The A / V input unit 104 may include a graphics processor (GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 106. The image frame processed by the graphics processor 1041 can be stored in the memory 109 (or other storage medium) or sent via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) via the microphone 1042 in a telephone call mode, a recording mode, a voice recognition mode, and other operating modes, and can process such sound into audio data. The processed audio (voice) data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 101 in the case of a telephone call mode. The microphone 1042 can implement various types of noise elimination (or suppression) algorithms to eliminate (or suppress) noise or interference generated in the process of receiving and sending audio signals.

[0080] The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Optionally, the light sensor includes an ambient light sensor and a proximity sensor. Optionally, the ambient light sensor can adjust the brightness of the display panel 1061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 1061 and / or the backlight when the mobile terminal 100 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured on the mobile phone, such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0081] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0082] The user input unit 107 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile terminal. Optionally, the user input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1071 or near the touch panel 1071 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Optionally, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 110, and can receive and execute the command sent by the processor 110. In addition, the touch panel 1071 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1071, the user input unit 107 may further include other input devices 1072. Optionally, the other input devices 1072 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, etc., which are not specifically limited here.

[0083] Optionally, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the processor 110 to determine the type of touch event. Then, the processor 110 provides corresponding visual output on the display panel 1061 according to the type of touch event. Figure 1 In the figure, the touch panel 1071 and the display panel 1061 are used as two independent components to implement the input and output functions of the mobile terminal. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to implement the input and output functions of the mobile terminal, which is not limited here.

[0084] The interface unit 108 serves as an interface through which at least one external device can be connected to the mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 108 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the mobile terminal 100 or may be used to transmit data between the mobile terminal 100 and an external device.

[0085] The memory 109 can be used to store software programs and various data. The memory 109 can mainly include a program storage area and a data storage area. Optionally, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 109 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0086] The processor 110 is the control center of the mobile terminal. It uses various interfaces and lines to connect various parts of the entire mobile terminal. It executes various functions of the mobile terminal and processes data by running or executing software programs and / or modules stored in the memory 109, and calling data stored in the memory 109, so as to monitor the mobile terminal as a whole. The processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor and a modem processor. Optionally, the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 110.

[0087] The mobile terminal 100 may also include a power supply 111 (such as a battery) for supplying power to various components. Preferably, the power supply 111 may be logically connected to the processor 110 via a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0088] although Figure 1 Not shown, the mobile terminal 100 may further include a Bluetooth module, etc., which will not be described in detail here.

[0089] In order to facilitate understanding of the embodiments of the present application, the communication network system on which the mobile terminal of the present application is based is described below.

[0090] See also Figure 2 , Figure 2 A communication network system architecture diagram is provided for an embodiment of the present application, wherein the communication network system is an LTE system of universal mobile communication technology, and the LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203 and an operator's IP service 204, which are connected in sequence for communication.

[0091] Optionally, UE201 may be the above-mentioned terminal 100, which will not be described in detail here.

[0092] E-UTRAN 202 includes eNodeB 2021 and other eNodeBs 2022 , etc. Optionally, eNodeB 2021 may be connected to other eNodeBs 2022 via a backhaul (eg, an X2 interface), and eNodeB 2021 is connected to EPC 203 , and eNodeB 2021 may provide UE 201 with access to EPC 203 .

[0093] EPC203 may include MME (Mobility Management Entity) 2031, HSS (Home Subscriber Server) 2032, other MMEs 2033, SGW (Serving Gate Way) 2034, PGW (PDN Gate Way) 2035 and PCRF (Policy and Charging Rules Function) 2036, etc. Optionally, MME 2031 is a control node that processes signaling between UE 201 and EPC 203, and provides bearer and connection management. HSS 2032 is used to provide some registers to manage functions such as home location register (not shown in the figure), and store some user-specific information such as service features and data rates. All user data can be sent through SGW2034, PGW2035 can provide IP address allocation and other functions for UE 201, PCRF2036 is the policy and charging control policy decision point for service data flow and IP bearer resources, which selects and provides available policy and charging control decisions for the policy and charging execution function unit (not shown in the figure).

[0094] The IP service 204 may include the Internet, an intranet, an IMS (IP Multimedia Subsystem) or other IP services.

[0095] Although the above introduction takes the LTE system as an example, those skilled in the art should know that the present application is not only applicable to the LTE system, but also to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA, 5G and future new network systems (such as 6G), etc., without limitation here.

[0096] Figure 3The hardware structure diagram of a controller 140 provided in the present application is shown in FIG. The controller 140 includes a memory 1401 and a processor 1402. The memory 1401 is used to store program instructions, and the processor 1402 is used to call the program instructions in the memory 1401 to execute the steps executed by the controller in the first embodiment of the above method. The implementation principle and beneficial effects are similar and will not be described in detail here.

[0097] Optionally, the controller further includes a communication interface 1403, which can be connected to the processor 1402 via a bus 1404. The processor 1402 can control the communication interface 1403 to implement the receiving and sending functions of the controller 140.

[0098] Figure 4 The hardware structure diagram of a network node 150 provided in the present application is shown in FIG. The network node 150 includes: a memory 1501 and a processor 1502, the memory 1501 is used to store program instructions, and the processor 1502 is used to call the program instructions in the memory 1501 to execute the steps performed by the first node in the first embodiment of the above method, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0099] Optionally, the controller further includes a communication interface 1503, which can be connected to the processor 1502 via a bus 1504. The processor 1502 can control the communication interface 1503 to implement the receiving and sending functions of the network node 150.

[0100] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the methods of various embodiments of the present application.

[0101] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a storage medium, or transmitted from one storage medium to another storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid state hard disk solid state disk, SSD), etc.

[0102] Based on the above-mentioned mobile terminal hardware structure and communication network system, various embodiments of the present application are proposed.

[0103] First embodiment

[0104] Reference Figure 5 , Figure 5 is a flow chart of a processing method according to the first embodiment. The processing method of the embodiment of the present application can be applied to a processing device, including:

[0105] Step S10: performing prediction processing on the current block according to at least one weight information to determine or generate a prediction block.

[0106] In this embodiment, the processing device may be a smart terminal, such as a mobile phone, a computer, etc., or a server, such as a local server or a cloud server. In this embodiment and this application, the processing device is mainly described as a smart terminal.

[0107] Optionally, the technical solution of this embodiment can be applied to the fields of image coding and decoding, video coding and decoding, hardware video coding and decoding, dedicated circuit video coding and decoding, real-time video coding and decoding, etc.

[0108] Optionally, the processing device may obtain video image data from a video source, segment each frame of the video image data to obtain a plurality of image blocks, and determine an image block to be predicted among the plurality of image blocks as the current block.

[0109] Optionally, the current block may be an image block to be predicted in an encoder and / or a decoder.

[0110] Optionally, the prediction block may be further interpolated to obtain an interpolated prediction block.

[0111] Optionally, the weight information may be information about the proportion of each reference block contributing to the final prediction result in the process of processing the current block in combination with the reference block of the current block. For example, the weight information may be a weight coefficient, or a weight vector or a weight matrix including the weight coefficient, etc., which indicates information used to measure the proportion of different elements (e.g., prediction values) in the result or decision.

[0112] Alternatively, the reference block may be a pixel block corresponding to the current block extracted from a reference frame (also referred to as a reference image) or a current frame (also referred to as a current image), the reference block may have the same size as the current block, and the reference block may be used as a prediction block for generating the current block. By using the reference block, the encoder may utilize temporal or spatial correlation to reduce redundant information in the current frame, thereby achieving efficient compression.

[0113] Optionally, a reference frame refers to an already encoded and reconstructed frame (also called an image), which may be a forward reference frame (i.e., a frame / image that precedes the current frame in the playback order) or a backward reference frame (i.e., a frame / image that follows the current frame in the playback order).

[0114] Optionally, performing prediction processing of the current block according to at least one weight information may include: intra-frame prediction processing and / or inter-frame prediction processing. Optionally, the prediction processing may also include: weighted processing. Optionally, weighted processing is a processing process in the intra-frame prediction processing and / or inter-frame prediction processing for fusing multiple pieces of information or pixel values ​​of the same type according to corresponding proportions indicated by the weight information to obtain a corresponding fusion result.

[0115] Optionally, the weight information can reflect the reliability and / or relevance of each reference block. The higher the reliability and / or relevance of the reference block (for example, a reference block with a small motion estimation error and a high texture matching degree), the larger the value in the corresponding weight information.

[0116] Optionally, weighted processing may be performed based on at least one reference block of the current block and weight information, and weighting refers to multiplying by a weight, that is, multiplying by a coefficient. By weighting, the weight information is applied to adjust the importance of the reference block, thereby generating a more accurate prediction result.

[0117] Optionally, the weight information may include: at least one of a weight coefficient, a weight vector and a weight matrix.

[0118] Optionally, the weight coefficient can be a scalar value, which can represent the weight of a single reference block. For example, the current block includes two reference blocks P1 and P2, and the weight coefficients corresponding to the reference blocks P1 and P2 are W1 and W2 respectively; and the predicted block P0 obtained by the final prediction can be expressed as: P0 = W1×P1+W2×P2.

[0119] Optionally, the weight vector can be a one-dimensional array containing multiple weight coefficients, each coefficient corresponding to a reference block or a prediction block. For example, the current block includes three reference blocks P1, P2 and P3, and the corresponding weight vector is w = [W1, W2, W3], and the predicted prediction block P0 can be expressed as:

[0120]

[0121] Optionally, the weight matrix can be a two-dimensional array, and each element in the weight matrix represents the weight coefficient of the pixel at a specific position, which is usually the same size as the current block. For example, the current block includes two reference blocks P1 and P2, and a weight matrix W[i, j, k], which represents the weight of the pixel point at the i-th row and i-th column on the k-th reference block (k=1 or 2), and each pixel of the predicted block P0 can be expressed as:

[0122] P0[i,j]=W[i,j,1]×P1[i,j]+W[i,j,2]×P2[i,j]+c;

[0123] Optionally, c is a bias term. Optionally, c is equal to 0.

[0124] Optionally, if the current block includes N reference blocks, the value of k is 1 to N. The i and j are integers greater than 0.

[0125] Optionally, refer to Figure 6When the processing device is an encoder on the encoding side, the encoder can receive video data input from a video source, such as receiving a video image from a video source, determining an image to be predicted in the video image, dividing the image to be predicted into multiple image blocks, and using the temporal and / or spatial correlation between the video images to perform prediction processing on each of the multiple image blocks, including intra-frame prediction processing and / or inter-frame prediction processing, and the intra-frame prediction processing and / or inter-frame prediction processing respectively include multiple prediction modes. For these prediction modes, the encoder uses, for example, rate-distortion cost to determine the prediction mode finally adopted by each of the multiple image blocks, for example, calculating the rate-distortion cost corresponding to each prediction mode or the rate-distortion cost of a combination of several prediction modes, so as to determine the minimum rate-distortion cost from the multiple rate-distortion costs, and the prediction mode or combination of prediction modes corresponding to the minimum rate-distortion cost is the prediction mode finally adopted by the image block. For example, the weighted processing is a fusion process of multiple prediction values ​​in the prediction mode finally adopted to obtain a prediction value by weighting multiple prediction values. For another example, the weighted processing is a fusion process of multiple prediction values ​​in a prediction mode finally adopted, in which multiple prediction values ​​obtained by multiple sub-modes in the prediction mode finally adopted are weighted to obtain one prediction value.

[0126] Optionally, the inter-frame prediction mode based on weight information in the embodiment of the present application can be used as an independent geometric partitioning mode to compete with other geometric partitioning modes in the encoder and / or decoder, and the final prediction mode is determined or obtained according to the rate-distortion cost of each geometric partitioning mode applied to the current block.

[0127] Optionally, when the prediction mode finally adopted is the inter-frame prediction mode based on weight information provided in this embodiment, at least one weight information can be determined or obtained through at least one reference area of ​​the current block, and the step flow of the above step S10 can be executed.

[0128] Optionally, refer to Figure 6, if based on step S10, after the prediction block of the current block is obtained, the residual block between the prediction block and the current block can be calculated; the residual block can be transformed and quantized, and then encoded by the entropy encoder to form an encoded bit stream. Optionally, the encoded bit stream can also include prediction parameters corresponding to the determined prediction mode and related auxiliary information (side information). Optionally, the prediction parameters are packaged into the encoded bit stream after entropy encoding. Optionally, the prediction parameters include indication information of the prediction mode. Optionally, the transformed and quantized residual block can be added to the corresponding prediction data (such as the prediction block) obtained by using the prediction mode after inverse quantization and inverse transformation to obtain a reconstructed block. After obtaining the reconstructed block, the loop filtering module performs loop filtering on the reconstructed block according to the filter control parameters to reduce distortion. After loop filtering, the reconstructed block after loop filtering is stored according to the coded image buffer.

[0129] Optionally, refer to Figure 7 , when the processing device is a decoder on the decoding side, after the decoder receives the encoded bit stream, the entropy decoding unit of the decoder will parse and decode the encoded bit stream to obtain the transform coefficients. The inverse transform unit and the inverse quantization unit of the decoder perform inverse transform and inverse quantization on the transform coefficients to obtain a residual block. Optionally, the entropy decoding unit of the decoder will parse and decode the encoded bit stream to obtain prediction data, such as prediction parameters and related auxiliary information. The prediction processing unit of the decoder performs prediction processing using the prediction parameters to determine the prediction block corresponding to the residual block. Optionally, the prediction processing includes intra-frame prediction processing and / or inter-frame prediction processing, and the intra-frame prediction processing and / or inter-frame prediction processing respectively include a combination of one or more prediction modes in a plurality of prediction modes. When the prediction parameter indicates that the corresponding prediction mode is the prediction mode provided in this embodiment, the prediction mode is the prediction mode corresponding to the obtained residual block.

[0130] Optionally, refer to Figure 7 When the prediction mode in this embodiment is adopted, the above step S10 is executed. After obtaining the prediction block of the current block, the obtained residual block and the corresponding prediction block are added to obtain a reconstructed block. The loop filtering unit in the decoder performs loop filtering on the reconstructed block according to the filter control data to reduce distortion and improve video quality. The reconstructed block after loop filtering is further combined into a decoded image and stored in a decoded image buffer or output as decoded video data.

[0131] Optionally, when inter-frame prediction is performed on the current block and the prediction mode provided in this embodiment is selected for prediction, the current block may be predicted according to at least one piece of weight information to obtain a predicted block of the current block.

[0132] Optionally, the geometric partitioning mode (GPM) is a prediction mode used by the next generation video compression standard (Versatile Video Coding, VVC) for the boundary part of the moving object in the image. Compared with other prediction modes, the GPM mode can divide the boundary of the moving object in the image more finely, by fitting the dividing line and the boundary of the moving object, dividing the edge coding unit (Code unit) of the moving object into non-rectangular sub-coding units for unidirectional prediction, thereby obtaining the prediction value of the entire coding unit.

[0133] Optionally, in the GPM mode, the numerical distribution of the weight information corresponds to different division methods. For example, the current block includes two reference blocks P1 and P2, and the corresponding weight matrices are W[i, j, 1] and W[i, j, 2]. The formula for determining each pixel of the prediction block P0 is: P0[i, j] = (W[i, j, 1] × P1[i, j] + W[i, j, 2] × P2[i, j]) / 16. Please refer to Figure 8 and Fig. 9 , Figure 8 is a schematic diagram of the weight matrix W[i, j, 1], Fig. 9 is a schematic diagram of the weight matrix W[i, j, 2]. The sum of the total weights of the weight matrix W[i, j, 1] and the weight matrix W[i, j, 2] is 16. Since the above formula divides the result of W[i, j, 1]×P1[i, j]+W[i, j, 2]×P2[i, j] by 16, the actual weight coefficients of P1[i, j] and P2[i, j] are W[i, j, 1] / 16 and W[i, j, 1] / 16, and W[i, j, 1] / 16+W[i, j, 1] / 16=1. By analyzing the physical meaning of the above formula, we can see that Figure 8 and Fig. 9The component with a value of 8 in the weight matrix corresponds to the position of the GPM dividing line. That is, the actual weight coefficient of P1[i, j] and P2[i, j] is 0.5, and the position of the component in the corresponding weight matrix corresponds to the position of the GPM dividing line. In this embodiment, the dividing line is a straight line at 45 degrees from the upper left corner. It should be noted that for other formulas for determining the prediction block P0, for example, P0[i, j] = (W[i, j, 1] × P1[i, j] + W[i, j, 2] × P2[i, j]) / 18, the values ​​of the weight matrices W[i, j, 1] and W[i, j, 2] corresponding to the straight line at 45 degrees from the upper left corner will be different. In this case, the components with a value of 4 in the weight matrices W[i, j, 1] and W[i, j, 2] correspond to the position of the GPM dividing line. Those skilled in the art can know how to correspond the numerical distribution of weight information to different division methods through the above examples. For the sake of brevity, it will not be repeated. Further, the present application is not limited thereto. In other embodiments, the numerical distribution of the weight matrix may also correspond to other dividing lines in the GPM mode. In other words, the numerical distribution of the weight matrix corresponds to different division methods. Optionally, the sum of the total weights may be set according to actual conditions, for example, the sum of the total weights may be set to 8, 16, 32, 64, etc., which is not limited in this embodiment.

[0134] Optionally, in addition to the division method corresponding to the straight line, the numerical distribution of the weight information can also correspond to other non-straight line division methods, such as broken lines, curves, etc. For example, please refer to Fig.10 and Fig.11 , assuming the total weight sum is 16. Similarly, Fig.10 and Fig.11 The component with a value of 8 in the weight matrix corresponds to the position of the GPM dividing line, and Fig.10 and Fig.11 The following are schematic diagrams of the weight matrix W[i, j, 1] and the weight matrix W[i, j, 2] corresponding to the polyline partitioning method. For another example, please refer to Fig.12 and Fig.13 , assuming the total weight sum is 16. Similarly, Fig.12 and Fig.13 The component with a value of 8 in the weight matrix corresponds to the position of the GPM dividing line, and Fig.12 and Fig.13 Schematic diagrams of the weight matrix W[i, j, 1] and the weight matrix W[i, j, 2] corresponding to the curve partitioning method.

[0135] Optionally, the numerical distribution of the weight information may also correspond to the case where there is no dividing line. Fig.14 and Fig.15 , assuming the total weight is 16, Fig.14 and Fig.15 There is no specific rule for the distribution of the components with a value of 8 in the weight matrix of Fig.14 and Fig.15 Schematic diagrams of the weight matrix W[i, j, 1] and the weight matrix W[i, j, 2] when there is no dividing line.

[0136] Optionally, the decoder can determine at least one weight information of the current block from the decoded encoded bit stream, perform prediction processing on the current block according to at least one weight information, and determine or generate a prediction block. In one embodiment, since the numerical distribution of the weight information can correspond to different division methods (for example, different division methods in the GPM mode), and in the case of no division, the prediction block corresponding to the current block can also be directly determined or generated according to the weight information, and then the bit stream does not need to encode the division method information, saving the signaling required to be transmitted by the division method. In this way, the implementation method of the present application reduces the signaling overhead and improves the compression efficiency of the video. And / or, since the boundaries of moving objects are not necessarily all straight line boundaries, the prediction accuracy of the weight information and the prediction accuracy of the pixels using the conventional linear division method will be affected to a certain extent. And / or, the angle and offset of the dividing line will be limited to a series of preset discrete values, so that the roughness of the linear division is further improved, affecting the prediction accuracy. In the implementation manner of the present application, the division method is no longer limited to linear division, and thus the influence of linear division on the prediction result can be effectively eliminated, thereby improving the flexibility and accuracy of video image prediction, and further improving the prediction effect during video encoding and / or decoding.

[0137] Optionally, when the processing device is an encoder, the prediction value obtained for the first time may be a prediction value obtained in a corresponding prediction mode, and the prediction value may be directly used in a rate-distortion cost process.

[0138] Optionally, when the processing device is a decoder, the prediction value obtained for the first time may be a prediction value obtained in a prediction mode corresponding to a block to be predicted (ie, an image block located in a decoding end) indicated by a syntax element obtained by parsing the bitstream.

[0139] Optionally, the prediction block can be used as the target image block, the residual block between the target image block and the current block is calculated, and then the entropy encoder is used to encode the target image block and the residual block between the target image block and the current block is calculated, and then the target image block is encoded by the entropy encoder through transformation and quantization to form an encoded bit stream. The prediction block can also be processed accordingly, for example, processed by other models, the processed image block is used as the target image block, and the step of calculating the residual block between the target image block and the current block and subsequent steps are performed.

[0140] Optionally, the prediction block is a first prediction block; a first prediction mode is determined or generated according to at least one first prediction block; and a prediction process is performed on the current block according to the first prediction mode to obtain at least one second prediction block.

[0141] Optionally, in the first prediction mode, a more complex interpolation filter (e.g., a high-order polynomial filter, an adaptive interpolation filter, etc.) can be used to process the prediction block to determine or generate a target image block (i.e., a second prediction block) to improve the match between the target image block and the original image block (i.e., the current block), especially when processing scenes with complex motion or edges.

[0142] Optionally, in the first prediction mode, the prediction block can be further predicted by the angle prediction mode; by taking into account the directional features in the image content (such as horizontal, vertical, diagonal, etc.), the direction that best reflects these features is selected for prediction to obtain the target image block. Optionally, the inter-frame prediction mode based on weight information of the embodiment of the present application can be fused with the angle prediction mode to generate a fused prediction value and a corresponding prediction block.

[0143] In this embodiment, the prediction processing of the current block can be realized according to at least one weight information, thereby enriching the prediction processing method; and the prediction result can be flexibly adjusted through the weight information, thereby improving the flexibility of video image prediction, and thus improving the prediction effect of video encoding and / or decoding. And / or because the numerical distribution of the weight information can correspond to different division methods, and in the case of no division, the prediction block corresponding to the current block can also be directly determined or generated according to the weight information, and then the encoding division method information is not required in the bit stream, saving the signaling required to transmit the division method, and improving the compression efficiency of the video.

[0144] Second embodiment

[0145] Based on the first embodiment, a second embodiment is proposed.

[0146] In this embodiment, the weight information is determined or obtained from at least one reference area of ​​the current block.

[0147] Optionally, at least one weight is determined or obtained according to at least one reference area of ​​the current block; and prediction processing is performed on the current block according to the at least one weight information to determine or generate a prediction block.

[0148] Optionally, the reference area may be an image area used to determine or obtain at least one weight information of the current block, and may be an image area adjacent to the current block, or an image area non-adjacent to the current block.

[0149] Alternatively, the reference region may be a region consisting of coded and reconstructed pixels or blocks from the same frame and / or different frames.

[0150] Optionally, the reference area may include at least one of a pixel, an image block, and an image area used to determine or obtain at least one weight information of the current block.

[0151] Optionally, the decoder may determine at least one weight information of the current block from at least one reference area determined by decoding the encoded bit stream, and perform prediction processing on the current block according to the at least one weight information to determine or generate a prediction block.

[0152] Optionally, since the numerical distribution of the weight information corresponds to different division methods, and in the absence of division, the prediction block corresponding to the current block can be directly determined or generated based on the weight information, and thus the division method information does not need to be encoded in the bit stream, saving the signaling required to be transmitted by the division method and improving the compression efficiency of the video.

[0153] In this embodiment, the current block is processed by the weight information determined by the reference area of ​​the current block, which can take into account the association between the current block and its reference area to improve the accuracy of the obtained weight information, thereby improving the prediction effect of the prediction block determined or generated based on the weight information, and further improving the prediction effect in the video encoding and / or decoding process.

[0154] Optionally, at least one weight information of the current block can be determined based on the reference area of ​​the current block at the encoding end or the decoding end, and the current block can be predicted based on the at least one weight information to determine or generate a prediction block. In one embodiment, since the numerical distribution of the weight information can correspond to different division methods (for example, different division methods in the GPM mode), and in the absence of division, the prediction block corresponding to the current block can also be directly determined or generated based on the weight information, and then the encoding division method information is not required in the bit stream, saving the signaling required to be transmitted by the division method. In this way, the implementation method of the present application reduces the signaling overhead and improves the compression efficiency of the video.

[0155] Optionally, the method for determining or obtaining the reference area includes at least one of the methods 1 to 6:

[0156] Method 1: determine or obtain at least one reference area based on at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels and upper left non-adjacent pixels of the current block.

[0157] Optionally, the upper adjacent pixel may be a pixel located above and adjacent to the current block in the same frame image.

[0158] Optionally, the upper non-adjacent pixel may be a pixel in the same frame image that is located above the current block but is not adjacent to the current block.

[0159] Optionally, the left adjacent pixel may be a pixel located on the left side and adjacent to the current block in the same frame image.

[0160] Optionally, the left non-adjacent pixel may be a pixel located on the left side of the current block but not adjacent to the current block in the same frame image.

[0161] Optionally, the upper left adjacent pixel may be a pixel located at the upper left of the current block and adjacent to the current block in the same frame image.

[0162] Optionally, the upper left non-adjacent pixel may be a pixel located at the upper left of the current block in the same frame image but not adjacent to the current block.

[0163] Optionally, at least one of the upper adjacent pixel, the upper non-adjacent pixel, the left adjacent pixel, the left non-adjacent pixel, the upper left adjacent pixel and the upper left non-adjacent pixel may be a reconstructed pixel or a predicted pixel.

[0164] Optionally, at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels and upper left non-adjacent pixels can be directly used as a reference area, or at least one acquired pixel can be deduced or calculated to obtain a reference area.

[0165] Optionally, the reference area of ​​the current block may be acquired according to a preset mapping / correspondence rule.

[0166] Optionally, a reference area may be selected from upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels, and upper left non-adjacent pixels of the current block according to a prediction model.

[0167] For example, refer to Fig.16 , the upper adjacent pixels of the current block are determined as reference area A; the left adjacent pixels of the current block are determined as reference area B; the upper left adjacent pixels of the current block are determined as reference area C. The upper adjacent pixels, the left adjacent pixels and the upper left adjacent pixels all include multiple pixel points.

[0168] Optionally, at least one reference area is determined or obtained based on at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels and upper left non-adjacent pixels of the current block; at least one weight information is determined or obtained based on the at least one reference area; and prediction processing is performed on the current block based on the at least one weight information to determine or generate a prediction block.

[0169] Optionally, if some locations lack valid pixel data, they can be filled with adjacent valid pixels.

[0170] In this embodiment, the reference area is determined or obtained according to at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels, and upper left non-adjacent pixels of the current block. Since the current block usually has a high similarity with at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels, and upper left non-adjacent pixels of the current block, the reference area determined according to these adjacent pixels and / or non-adjacent pixels can improve the prediction accuracy of the weight information in the prediction process, thereby further improving the prediction effect of the current block.

[0171] Method 2, determining or obtaining at least one reference area according to at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block and a default block corresponding to the current block;

[0172] Optionally, at least one of a neighbor block, a non-neighbor block, a co-located block, a time domain block and a default block corresponding to the current block is used as at least one reference area.

[0173] Optionally, at least one reference area is determined or obtained according to image block information of at least one of a neighbor block, a non-neighbor block, a co-located block, a time domain block and a default block corresponding to the current block.

[0174] Optionally, the image block information may include at least one of a block size, a block area, an image block attribute and an image block type.

[0175] Optionally, the block size includes the width, height, ratio, depth, area, resolution and number of pixels of the block, etc. The image block attributes may include the position and / or image texture of the block. The image block type may include a natural image or a screen content image, etc.

[0176] Optionally, at least one reference area can be determined or obtained based on at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels and upper left non-adjacent pixels of at least one of the neighboring blocks, non-neighboring blocks, co-located blocks, time domain blocks and default blocks corresponding to the current block.

[0177] Optionally, at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels and upper left non-adjacent pixels of at least one of the neighboring blocks, non-neighboring blocks, co-located blocks, time domain blocks and default blocks corresponding to the current block can be used as at least one reference area.

[0178] Optionally, at least one reference area may be determined or obtained based on at least one of width, height, block size and block area of ​​at least one of a neighbor block, a non-neighbor block, a co-located block, a time domain block and a default block corresponding to the current block.

[0179] Optionally, the neighbor block may be a block adjacent to the current block, and may be a block that has been predicted or reconstructed.

[0180] Alternatively, the non-neighbor block may be a block that is not adjacent to the current block, and may be a block that has been predicted or reconstructed.

[0181] Optionally, the co-located block may be an image block in a co-located image, which has the same position and size as the current block, and the co-located image may be an image in the reference image that is closest to the current image in terms of time.

[0182] Optionally, the time domain block may be a block distinguished from the time domain, such as an image block in other frames before or after the current frame. For example, if there is video data containing a first frame image, a second frame image, and a third frame image played at the first second, the second second, and the third second, respectively, and the current block is a block divided from the second frame image, then it can be determined that the time domain block corresponding to the current block is a corresponding image block in other frames except the second frame.

[0183] Optionally, the default block may be a block set in advance, for example, a block with typical pixel features pre-set by an encoder and / or a decoder.

[0184] Optionally, at least one reference area is determined or obtained based on at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block and a default block corresponding to the current block; at least one weight information is determined or obtained based on the at least one reference area; and the current block is predicted based on the at least one weight information to determine or generate a prediction block.

[0185] In this embodiment, at least one reference area is determined or obtained based on at least one of the neighboring blocks, non-neighboring blocks, co-located blocks, time domain blocks and default blocks corresponding to the current block, thereby ensuring a close association between the determined or obtained reference area and the current block, thereby making the subsequent weight information determined or obtained based on the at least one reference area more accurate and further improving the prediction effect on the current block.

[0186] Method three, determining or obtaining at least one reference area according to at least one of the width, height, block size and block area of ​​the current block;

[0187] Optionally, at least one of pixels, image blocks and image regions in at least one reference region is determined or obtained based on at least one of width, height, block size and block area of ​​the current block.

[0188] Optionally, at least one reference area may be determined or obtained according to the width, height and first mapping table of the current block. Optionally, the first mapping table may be as shown in Table 1 below:

[0189] Table 1

[0190] Width, Height Size of the reference area <![CDATA[Either the width or the height of the current block is less than X1]]> <![CDATA[Dimension A1]]> <![CDATA[The width and height of the current block are both greater than or equal to X1, and either the width or the height is less than X2]]> <![CDATA[Dimension A2]]> <![CDATA[The width and height of the current block are both greater than or equal to X2]]> <![CDATA[Size A3]]> <![CDATA[The width and height of the current block are both greater than or equal to X2, and either the width or the height is less than X3]]> <![CDATA[Size A4]]>

[0191] Optionally, the height of at least one reference region is equal to a first preset multiple of the height of the current block. Optionally, the width of at least one reference region is equal to a second preset multiple of the width of the current block.

[0192] Optionally, the reference area or the image area in the reference area may be an encoded area or a decoded area.

[0193] Optionally, the coded area or the decoded area can be determined by the position of the upper left corner pixel, the height and width of the coded area or the decoded area. For example, the position of the upper left corner pixel can be the position of the upper left corner position of the current block N (N is greater than 1) times the height of the image block and N times the width of the image block to the left. The width of the coded area or the decoded area is an integer multiple of the width of the current block, and the height of the coded area or the decoded area is an integer multiple of the height of the current block.

[0194] Optionally, the reference area may be determined according to the block size of the current block and the second mapping table. Optionally, the second mapping table may be as shown in Table 2 below:

[0195] Table 2

[0196] Block size Size of the reference area <![CDATA[The block size of the current block is smaller than X4]]> <![CDATA[Dimension B1]]> <![CDATA[The block size of the current block is greater than X4 and less than X5]]> <![CDATA[Dimension B2]]> <![CDATA[The block size of the current block is greater than X5 and less than X6]]> <![CDATA[Dimension B3]]> <![CDATA[The block size of the current block is greater than X6 and less than X7]]> <![CDATA[Size B4]]>

[0197] Optionally, the block size includes at least one of width, height, ratio, depth, area, resolution, and number of pixels of the block. Optionally, X4 to X7 may be preset thresholds corresponding to at least one of width, height, ratio, depth, area, resolution, and number of pixels of the block size.

[0198] Optionally, the reference area may be determined according to the block area of ​​the current block and a third mapping table. Optionally, the third mapping table may be as shown in Table 3 below:

[0199] Table 3

[0200] Block area Area of ​​reference area <![CDATA[The block area of the current block is less than X8]]> <![CDATA[Area C1]]> <![CDATA[The block area of the current block is greater than X8 and less than X9]]> <![CDATA[Area C2]]> <![CDATA[The block area of the current block is greater than X9 and less than X 10 > <![CDATA[Area C3]]> <![CDATA[The block area of the current block is greater than X 10 and less than X 11 > <![CDATA[Area C4]]>

[0201] Optionally, the position of the reference area may be determined or obtained according to the upper adjacent pixels, the left adjacent pixels and the upper left adjacent pixels of the current block, and then the size of the reference area may be determined according to the current block width and height and the first mapping table.

[0202] Exemplarily, the reference area of ​​the current block includes: a first area adjacent to the top of the current block, a second area adjacent to the left of the current block, and a third area adjacent to the top left of the current block. The width of the first area is equal to twice the width of the current block, and the height of the second area is equal to twice the height of the current block. If the width and height of the current block are both greater than or equal to 8, the height of the first area is 8, the width of the second area is 8, and the width and height of the third area are both 8. If any one of the width and height of the current block is less than 8, the height of the first area is 4, the width of the second area is 4, and the width and height of the third area are both 4.

[0203] Optionally, at least one reference area is determined or obtained based on at least one of the width, height, block size and block area of ​​the current block; at least one weight information is determined or obtained based on the at least one reference area; and the current block is predicted based on the at least one weight information to determine or generate a prediction block.

[0204] In this embodiment, by determining or obtaining at least one reference area based on at least one of the width, height, block size and block area of ​​the current block, it is ensured that the determined or obtained reference area is closely related to the current block, thereby making the subsequent weight information determined or obtained based on the at least one reference area more accurate, thereby improving the prediction effect on the current block.

[0205] Method 4: Determine or obtain at least one reference area based on a candidate block determined or obtained according to a candidate motion vector or a candidate block vector of a current block.

[0206] Optionally, a candidate block may be determined or obtained according to a plurality of candidate motion vectors or a plurality of candidate block vectors in a candidate list of the current block, and used as at least one reference area.

[0207] Optionally, the candidate block may be determined or obtained according to the candidate motion vector or the candidate block vector of the current block; at least one reference area may be determined or obtained according to at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels, and upper left non-adjacent pixels of the candidate block. The specific implementation process may refer to the solution in the above-mentioned method 1, that is, the current block in the method 1 may be replaced with the candidate block, which will not be repeated here.

[0208] Optionally, the candidate block can be determined or obtained based on the candidate motion vector or candidate block vector of the current block; and at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels and upper left non-adjacent pixels of the candidate block is used as at least one reference area.

[0209] Optionally, the candidate block can be determined or obtained according to the candidate motion vector or candidate block vector of the current block; at least one reference area can be determined or obtained according to at least one of the width, height, block size and block area of ​​the candidate block. The specific implementation process can refer to the solution in the above-mentioned method 3, that is, the current block in the method 3 can be replaced by the candidate block, which will not be repeated here.

[0210] Optionally, the candidate block may be determined or obtained based on the candidate motion vector or candidate block vector of the current block; at least one reference area may be determined or obtained based on at least one of a neighbor block, a non-neighbor block, a co-located block, a time domain block, and a default block corresponding to the candidate block. The specific implementation process may refer to the solution in the above-mentioned method 2, that is, the current block in the method 2 may be replaced with the candidate block, which will not be repeated here.

[0211] Optionally, the candidate block can be determined or obtained based on the candidate motion vector or candidate block vector of the current block; at least one of the neighbor block, non-neighbor block, co-located block, time domain block and default block corresponding to the candidate block is used as at least one reference area.

[0212] For example, refer to Fig.17 , determine or obtain the best matching block from the current image according to the best block vector (BV, Block Vector) of the current block; determine the upper adjacent pixels of the best matching block as the reference area A; determine the left adjacent pixels of the best matching block as the reference area B; determine the upper left adjacent pixels of the best matching block as the reference area C.

[0213] For example, refer to Fig.18 , determine or obtain a candidate block from a reference image according to a motion vector (MV) of the current block; determine the upper adjacent pixels of the candidate block as a reference area A; determine the left adjacent pixels of the candidate block as a reference area B; determine the upper left adjacent pixels of the candidate block as a reference area C.

[0214] Optionally, at least one reference area is determined or obtained based on the candidate block determined or obtained according to the candidate motion vector or candidate block vector of the current block; at least one weight information is determined or obtained according to the at least one reference area; and the current block is predicted based on the at least one weight information to determine or generate a prediction block.

[0215] In this embodiment, the candidate block determined or obtained based on the candidate motion vector or candidate block vector of the current block ensures that the determined or obtained reference area is closely related to the current block, thereby making the subsequent weight information determined or obtained based on at least one reference area more accurate and further improving the prediction effect for the current block.

[0216] Mode 5: If the first information of the current block satisfies the first condition, the reference area is the first reference area;

[0217] Optionally, the first reference region is Figures 16 to 18 Any one of the reference area A, reference area B, and reference area C shown in .

[0218] Optionally, the first information may be at least one of flag information of the current block, syntax elements, indication information, an index of a list, an aspect ratio of the current block, a width-height value range, and an area value range.

[0219] Optionally, flag information can be used to indicate a specific state or option, for example, to control the behavior of an encoder or decoder.

[0220] Optionally, the syntax element refers to a series of parameters and data structures defined in a video coding standard, and is used to describe specific coding information of video frames and blocks.

[0221] Optionally, the indication information is information used to indicate a certain state or condition, and may include flag information or more complex syntax elements.

[0222] Optionally, an index of a list refers to an integer value used to identify a specific item in a list. These lists may contain candidate motion vectors, reference frame indices, transform coefficients, etc. The index can be used to quickly find and access specific items in the list.

[0223] Optionally, the first information may be at least one of methods one to four, that is, the first information may be determining or obtaining at least one reference area based on at least one of the upper adjacent pixels, upper non-adjacent pixels, left adjacent pixels, left non-adjacent pixels, upper left adjacent pixels and upper left non-adjacent pixels of the current block; may be determining or obtaining at least one reference area based on at least one of the neighboring blocks, non-neighboring blocks, co-located blocks, time domain blocks and default blocks corresponding to the current block; may be determining or obtaining at least one reference area based on at least one of the width, height, block size and block area of ​​the current block; may be determining or obtaining at least one reference area based on a candidate block determined or obtained according to a candidate motion vector or candidate block vector of the current block.

[0224] Optionally, the first condition may be a condition set in advance and may be set according to user needs, and the first condition is not fixed and may be adaptively adjusted according to different scenarios.

[0225] Optionally, if the processing device detects that the current block satisfies the first condition, the first information may be selected for calculation to determine or obtain at least one reference area.

[0226] Optionally, the current block satisfies the first condition, including at least one of the following:

[0227] The value of the first information is a first value;

[0228] The value of the first information is within a first numerical range;

[0229] The value of the first information is greater than or equal to the first threshold;

[0230] The value of the first information is less than or equal to the second threshold.

[0231] Optionally, the value of the first information may be at least one of the width, height, block size and block area of ​​the current block.

[0232] Optionally, the first value may be a value of at least one of a width, a height, a block size and a block area of ​​the image block set in advance.

[0233] Optionally, the first numerical interval may be a numerical interval of at least one of a width, a height, a block size and a block area of ​​the image block set in advance.

[0234] Optionally, the first threshold may be a threshold of at least one of width, height, block size and block area of ​​the image block set in advance.

[0235] Optionally, the second threshold may be a threshold of at least one of width, height, block size and block area of ​​the image block set in advance.

[0236] Optionally, when the processing device detects that the first information of the current block satisfies the first condition, it may select the first information for calculation to determine or obtain at least one reference area.

[0237] Optionally, the first reference area may be a reference area that satisfies at least one of the first to fourth methods.

[0238] Optionally, if the first information of the current block satisfies the first condition, the reference area is the first reference area; at least one weight information is determined or obtained based on at least one reference area; and the current block is predicted based on the at least one weight information to determine or generate a prediction block.

[0239] In this embodiment, by selecting a reference area based on the first information, the similarity between the reference area and the current block (for example, the similarity of texture features, the similarity of pixel features, etc.) can be improved, so that the weight information determined or generated based on the reference area with high similarity to the current block has higher accuracy, further improving the prediction effect of the prediction block determined or generated according to the weight information, as well as the prediction effect in the video encoding and / or decoding process.

[0240] Various types of information of the current block are associated to ensure that the prediction accuracy of the current block will not be affected by changes in the information of the current block, so as to improve the prediction processing effect for the current block.

[0241] Method six: if the first information of the current block does not satisfy the first condition, the reference area is the second reference area.

[0242] Optionally, the first reference region and the second reference region are different.

[0243] Optionally, the first reference area may be a portion of at least one of a left adjacent area, an upper adjacent area and an upper left adjacent area of ​​the current block, and the second reference area may be an area of ​​the above areas different from the first reference area.

[0244] Optionally, if the first information of the current block does not meet the first condition, the reference area is the second reference area; at least one weight information is determined or obtained based on at least one reference area; and the current block is predicted based on the at least one weight information to determine or generate a prediction block.

[0245] In this embodiment, by selecting a reference area based on the first information, the similarity between the reference area and the current block (for example, the similarity of texture features, the similarity of pixel features, etc.) can be improved, so that the weight information determined or generated based on the reference area with high similarity to the current block has higher accuracy, further improving the prediction effect of the prediction block determined or generated according to the weight information, as well as the prediction effect in the video encoding and / or decoding process.

[0246] Third embodiment

[0247] Based on any of the above embodiments, a third embodiment is proposed.

[0248] In this embodiment, step S10 includes step S11 and step S12:

[0249] Step S11, determining or obtaining at least one weight information according to pixels of at least one reference area of ​​the current block and the first model;

[0250] Optionally, the at least one reference area in this embodiment may be at least one reference area determined according to at least one of the first to sixth methods in the second embodiment.

[0251] Optionally, the pixel of the reference area may be at least one pixel included in the reference area.

[0252] Optionally, pixels of at least one reference area of ​​the current block can be input into the first model, and at least one weight information can be determined or obtained based on the pixel characteristics of the reference area by the first model, so that the weights corresponding to each pixel of the current block can be adjusted according to its specific characteristics, thereby capturing changes more accurately.

[0253] Step S12: determining or generating a prediction block of the current block according to at least one piece of weight information and at least one reference block of the current block.

[0254] Optionally, weighted processing is performed according to at least one piece of weight information and pixels of at least one reference block of the current block to determine or generate a prediction block of the current block.

[0255] Optionally, the weight information includes a weight coefficient, and weighted processing is performed according to at least one weight coefficient and pixels of at least one reference block of the current block to determine or obtain at least one predicted pixel. Optionally, a predicted block of the current block is determined or generated according to the obtained multiple predicted pixels.

[0256] Optionally, the weight information includes a weight vector, and weighted processing is performed according to at least one weight vector and pixels of at least one reference block of the current block to determine or obtain at least one predicted pixel. Optionally, a predicted block of the current block is determined or generated according to the obtained multiple predicted pixels.

[0257] Optionally, the weight information includes a weight matrix, and weighted processing is performed according to the weight matrix and pixels of at least one reference block of the current block to determine or generate a prediction block of the current block.

[0258] Optionally, when the processing device is an encoder on the encoding side, if the prediction mode finally adopted by the encoder is the inter-frame prediction mode based on weight information provided in this embodiment, the encoder can divide the current block by the target division method determined or obtained by the first model, and obtain the weight information corresponding to the division method. And / or, the encoder can determine the motion vector corresponding to the current block, and determine or obtain at least one reference block of the current block. In one embodiment, the encoder determines or obtains at least one weight information based on the pixels of at least one reference area of ​​the current block and the first model, and determines or generates the prediction block of the current block based on the at least one weight information and at least one reference block of the current block. Determine or obtain the corresponding residual based on the original pixels corresponding to the current block and the predicted pixels of the prediction block, so as to encode the residual and determine or obtain the encoded bit stream; the encoded bit stream may include indication information of the prediction mode and the motion vector index or other motion information corresponding to the reference block. Optionally, the at least one weight information is the weight information corresponding to the division method. For example, Figures 8 to 13The weight matrix or other weight coefficients, weight vectors shown. Optionally, the numerical distribution of the weight information may also correspond to a case where there is no specific division method; since in the case of no division, the prediction block corresponding to the current block may also be directly determined or generated according to the weight information, and thus the division method information does not need to be encoded in the bitstream, thus saving the signaling required to transmit the division method.

[0259] Optionally, the target partitioning method may include: partitioning by at least one of a straight line, an oblique line, a curve, a rectangle, a circle, an ellipse, etc., and the sub-regions obtained by the partitioning may be regular matrices and / or irregular matrices. Optionally, the target partitioning method may also include: at least one of fixed-size partitioning (e.g., macroblocks, coding tree units, etc.), adaptive partitioning (e.g., quadtree partitioning, binary tree partitioning, ternary tree partitioning, etc.), geometric partitioning mode (Geometric Partitioning Mode, GPM) (e.g., 45-degree oblique line partitioning, horizontal partitioning, vertical partitioning, polygonal partitioning, etc.), curve partitioning and mixed partitioning.

[0260] Optionally, when the processing device is a decoder on the decoding side, the decoder can determine the prediction mode adopted by the encoder from the encoded bit stream. When the prediction mode finally adopted by the encoder is the inter-frame prediction mode based on weight information provided in this embodiment, the decoder determines or obtains at least one weight information based on the pixels of at least one reference area of ​​the current block and the first model; the decoder determines or generates a prediction block of the current block based on the at least one weight information and at least one reference block of the current block determined from the encoded bit stream. The decoder determines the residual corresponding to the prediction block based on the syntax elements obtained from the bit stream. The decoder determines or obtains the reconstructed block based on the residual and the prediction block to achieve decoding.

[0261] It can be understood that in the prediction mode of this embodiment, the processing device can determine or obtain the target division method of the current block through the first model. Therefore, the signaling required for transmitting the division method between the encoder and the decoder can be saved, thereby reducing the signaling overhead and improving the compression efficiency of the video.

[0262] Optionally, since the boundaries of moving objects are not necessarily all straight line boundaries, the accuracy of the straight line division method will be affected to a certain extent; and / or the angle and offset of the dividing line will be limited to a series of preset discrete values, so that the roughness of the linear division is further improved, thereby affecting the prediction accuracy. The method of the embodiment of the present application does not limit the block division method, thereby effectively reducing the impact of block division on the block prediction result and improving the prediction effect.

[0263] In this embodiment, by using the first model to directly perform weight prediction on the current block using the pixels of the reference area of ​​the current block, the influence of conventional linear division on the prediction accuracy can be avoided, and the correlation between the current block and its reference area can be taken into account, thereby improving the video encoding and / or decoding prediction effect. And / or, the signaling required for the division method transmission can be effectively saved, the signaling overhead can be reduced, and the video compression efficiency can be improved.

[0264] Optionally, at least one reference block of the current block is determined or obtained according to an index obtained from a bitstream.

[0265] Optionally, when the processing device is a decoder on the decoding side, the decoder can obtain a motion vector candidate index from the bitstream, and determine or obtain at least one reference block of the current block based on the motion vector candidate index. The decoder can determine the prediction mode from the bitstream, and if the prediction mode is the inter-frame prediction mode based on weight information provided in this embodiment, the encoder determines or obtains at least one weight information based on pixels of at least one reference area of ​​the current block and the first model; and determines or generates a prediction block of the current block based on the at least one weight information and at least one reference block of the current block.

[0266] Optionally, when the processing device is a decoder on the decoding side, the decoder can quickly determine or obtain at least one reference block from an index obtained from the bitstream, thereby performing decoding and improving decoding efficiency.

[0267] Optionally, the first model is determined or obtained according to at least one of the following methods 7 to 8:

[0268] Method seven: at least one of the width, height, size and area of ​​at least one reference region of the current block.

[0269] Optionally, a first model corresponding to the target model type is determined from a plurality of model types according to at least one of width, height, size and area of ​​at least one reference region of the current block.

[0270] Optionally, based on at least one of the geometric properties of at least one reference area of ​​the current block (i.e., width, height, block size and block area), a model type that is most suitable for processing these features is selected so that the first model can more accurately reflect the characteristics of the input data and improve the prediction effect.

[0271] Optionally, the structure of the first model is determined according to at least one of width, height, size and area of ​​at least one reference region of the current block.

[0272] Optionally, the specific structure of the selected model is adjusted according to at least one of the geometric properties (i.e., width, height, block size, and block area) of at least one reference area of ​​the current block. For example, when the first model belongs to a neural network model, the number of hidden layers (e.g., convolutional layers, fully connected layers), filter size, pooling strategy, etc. can be adjusted according to at least one of the width, height, size, and area of ​​at least one reference area. For example, when the first model belongs to a mathematical model based on a lookup table, the corresponding weight mapping can be pre-calculated and stored according to at least one of the width, height, size, and area of ​​at least one reference area. For example, when the first model belongs to a mathematical model based on a matrix, a specific transformation matrix can be designed to adapt to the above-mentioned geometric properties according to at least one of the width, height, size, and area of ​​at least one reference area.

[0273] Optionally, the structure of the first model may be determined according to the width and height of at least one reference area of ​​the current block and a fourth mapping table.

[0274] Optionally, the fourth mapping table may be as shown in Table 4 below:

[0275] Table 4

[0276]

[0277]

[0278] Optionally, the structure of the first model may be determined according to the block size of at least one reference area of ​​the current block and a fifth mapping table. Optionally, the fifth mapping table may be as shown in Table 5 below:

[0279] Table 5

[0280] Block size First Model <![CDATA[The block size of the reference area is smaller than X4]]> First structure <![CDATA[The block size of the reference area is greater than X4 and less than X5]]> Second structure <![CDATA[The block size of the reference area is greater than X5 and less than X6]]> The third structure <![CDATA[The block size of the reference area is greater than X6 and less than X7]]> The fourth structure

[0281] Optionally, the block size includes at least one of width, height, ratio, depth, area, resolution, and number of pixels of the block. Optionally, X4 to X7 may be preset thresholds corresponding to at least one of width, height, ratio, depth, area, resolution, and number of pixels of the block size.

[0282] Optionally, the structure of the first model may be determined according to the block size of at least one reference area of ​​the current block and a sixth mapping table. Optionally, the sixth mapping table may be as shown in Table 6 below:

[0283] Table 6

[0284] Block area First Model <![CDATA[The block area of the reference region is less than X8]]> First structure <![CDATA[The block area of the reference region is greater than X8 and less than X9]]> Second structure <![CDATA[The block area of the reference region is greater than X9 and less than X 10 > The third structure <![CDATA[The block area of the reference region is greater than X 10 and less than X 11 > The fourth structure

[0285] In this embodiment, based on at least one item of the width, height, size and area of ​​at least one reference region of the current block, a model that is most suitable for processing these features is determined or obtained, so that the first model can more accurately reflect the characteristics of the input data, improve the prediction effect of the weight information, and thereby improve the prediction effect in the video encoding and / or decoding process.

[0286] Optionally, if the width, height, size or area of ​​the reference area of ​​the current block does not match any of the multiple target model types, the reference area corresponding to the current block is transposed or downsampled to be converted into the width, height, size or area of ​​the reference area that matches at least one of the multiple target model types. In this way, more reference areas can be matched to the target model or target model type by transposing or downsampling, thereby reducing the number of target models or target model types as a whole.

[0287] Mode eight: at least one of the width, height, block size and block area of ​​the current block.

[0288] Optionally, there is an association between at least one of the width, height, block size and block area of ​​the current block and at least one reference area of ​​the current block. For example, at least one reference area can be determined or obtained based on at least one of the width, height, block size and block area of ​​the current block. Therefore, the first model can be determined or obtained through at least one of the width, height, block size and block area of ​​the current block.

[0289] According to at least one of the width, height, block size and block area of ​​the current block, a first model corresponding to the target model type is determined from multiple model types.

[0290] Optionally, based on at least one of the geometric properties of the current block (i.e., width, height, block size and block area), a model type that is most suitable for processing these features is selected so that the first model can more accurately reflect the characteristics of the input data and improve the prediction effect.

[0291] Optionally, the structure of the first model is determined according to at least one of a width, a height, a block size and a block area of ​​the current block.

[0292] Optionally, the specific structure of the selected model is adjusted according to at least one of the geometric properties of the current block (i.e., width, height, block size, and block area). For example, when the first model belongs to a neural network model, the number of hidden layers (e.g., convolutional layers, fully connected layers), filter size, pooling strategy, etc. can be adjusted according to at least one of the width, height, block size, and block area. For example, when the first model belongs to a mathematical model based on a lookup table, the corresponding weight mapping can be pre-calculated and stored according to at least one of the width, height, block size, and block area. For example, when the first model belongs to a mathematical model based on a matrix, a specific transformation matrix can be designed to adapt to the above-mentioned geometric properties according to at least one of the width, height, block size, and block area.

[0293] Optionally, the structure of the first model may be determined according to the width, height and seventh mapping table of the current block. Optionally, the seventh mapping table may be as shown in Table 7 below:

[0294] Table 7

[0295] Width, Height First Model <![CDATA[Either the width or the height of the current block is less than X1]]> First structure <![CDATA[The width and height of the current block are both greater than or equal to X1, and either the width or the height is less than X2]]> Second structure <![CDATA[The width and height of the current block are both greater than or equal to X2]]> The third structure <![CDATA[The width and height of the current block are both greater than or equal to X2, and either the width or the height is less than X3]]> The fourth structure

[0296] Optionally, the structure of the first model may be determined according to the block size of the current block and the eighth mapping table. Optionally, the eighth mapping table may be as shown in Table 8 below:

[0297] Table 8

[0298] Block size First Model <![CDATA[The block size of the current block is smaller than X4]]> First structure <![CDATA[The block size of the current block is greater than X4 and less than X5]]> Second structure <![CDATA[The block size of the current block is greater than X5 and less than X6]]> The third structure <![CDATA[The block size of the current block is greater than X6 and less than X7]]> The fourth structure

[0299] Optionally, the structure of the first model may be determined according to the block size of the current block and the ninth mapping table. Optionally, the ninth mapping table may be as shown in Table 9 below:

[0300] Table 9

[0301] Block area First Model <![CDATA[The block area of the current block is less than X8]]> First structure <![CDATA[The block area of the current block is greater than X8 and less than X9]]> Second structure <![CDATA[The block area of the current block is greater than X9 and less than X 10 > The third structure <![CDATA[The block area of the current block is greater than X 10 and less than X 11 > The fourth structure

[0302] In this embodiment, based on at least one of the width, height, block size and block area of ​​the current block, a model that is most suitable for processing these features is determined or obtained, so that the first model can more accurately reflect the characteristics of the input data, improve the prediction effect of the weight information, and thus improve the prediction effect in the video encoding and / or decoding process.

[0303] Fourth embodiment

[0304] Based on any of the above embodiments, a fourth embodiment is proposed.

[0305] In this embodiment, at least one piece of weight information is determined or obtained according to at least one of the following methods 9 to 13:

[0306] Method nine, pixels and neural network model of at least one reference area of ​​the current block.

[0307] Optionally, pixels of at least one reference area of ​​the current block may be used as input of the neural network model, and the association relationship between the pixels of the current block and its reference area learned by the neural network model may be fully utilized to output at least one weight information.

[0308] Optionally, at least one weight information is determined or obtained based on pixels of at least one reference area of ​​the current block and a neural network model, and a prediction block of the current block is determined or generated based on the at least one weight information and at least one reference block of the current block.

[0309] Optionally, a neural network model is a computational model that mimics the way neurons in the human brain process information, and it performs various tasks by learning complex patterns and features in data. A neural network model consists of multiple layers of neurons, including an input layer, at least one hidden layer, and an output layer. Each neuron receives an input signal, performs a weighted sum, and then generates an output through a nonlinear activation function. The weights of these connected neurons determine the degree of influence of the input signal on the output, while the bias is used to adjust the activation threshold of the neuron. Activation functions, such as ReLU, Sigmoid, or Tanh, give neural networks the ability to model nonlinearly, enabling them to solve problems that linear models cannot solve.

[0310] Optionally, the neural network model may include: an input layer, a hidden layer, a dropout layer, and an output layer.

[0311] Optionally, the input layer is the first layer that receives input data. In this layer, each node (or neuron) usually represents the characteristics of the input data. Hidden Layers are located between the input layer and the output layer. There can be multiple hidden layers, which process the input data in a weighted manner. Hidden layers include at least one of: fully connected layer, convolutional layer, pooling layer, recurrent layer and dropout layer. Fully connected layer: Each neuron is connected to each neuron in the previous layer. Convolutional layer: used to extract local features in the input data, often used in image processing. Pooling layer: used for downsampling, reducing the amount of data and calculation. Recurrent layer: such as LSTM or GRU, processing sequence data. Dropout layer: used to randomly "drop" (i.e. temporarily set the output to zero) a part of the neurons during model training. This can reduce the complex co-adaptation relationship between neurons, improve the generalization ability of the model, and reduce the risk of overfitting. Dropout layers can be added between multiple fully connected layers to prevent the neurons in these layers from being overly dependent on the specific output of the previous layer, thereby enhancing the generalization ability of the model. Dropout layers can be added between convolutional layers and fully connected layers to help reduce overfitting introduced by fully connected layers. The output layer is the layer that generates the final prediction result.

[0312] In this embodiment, the neural network model is used to directly predict the weight of the current block using the pixels of the reference area of ​​the current block, which can avoid the influence of conventional linear division on the prediction accuracy and take into account the correlation between the current block and its reference area, thereby improving the prediction effect of inter-frame prediction. And / or, it can also effectively save the signaling required for the division method to transmit, reduce the signaling overhead, and improve the compression efficiency of the video.

[0313] Optionally, determining or obtaining at least one weight information according to pixels of at least one reference area of ​​the current block and a neural network model includes:

[0314] Determine or obtain a first neural network model from a set of neural network models;

[0315] At least one weight information is determined or obtained based on pixels of at least one reference area of ​​the current block and the first neural network model.

[0316] Optionally, a first neural network model is determined or obtained from a group of pre-trained neural network models; pixels of at least one reference area of ​​the current block are used as input to the first neural network model, and the first neural network model outputs at least one weight information.

[0317] Optionally, the first neural network model can be determined or obtained from a group of neural network models through at least one reference area of ​​the current block, and a more matching first neural network model can be selected according to the characteristics of at least one reference area of ​​the current block.

[0318] Optionally, a first neural network model is determined or obtained from a group of neural network models based on at least one reference area of ​​the current block and a preset model mapping / correspondence rule.

[0319] Optionally, a set of neural network models are pre-selected and trained, which may include: at least one of a neural network model based on a fully connected layer, a neural network based on a convolutional layer, and a neural network based on a hybrid convolutional and fully connected layer.

[0320] Optionally, a neural network model based on fully connected layers is also called a multilayer perceptron (MLP); the model includes: an input layer, one or more hidden layers (i.e., fully connected layers), and an output layer, each layer contains a certain number of neurons, and all neurons between two adjacent layers are interconnected, i.e., "fully connected" between layers. Optionally, a dropout layer may be provided between at least one group of fully connected layers of the neural network model based on fully connected layers to help reduce overfitting introduced by the fully connected layers.

[0321] Optionally, pixels of at least one reference area of ​​the current block are fully connected according to at least one fully connected layer of the neural network model, for example, including linear transformation and activation function processing, so as to extract higher-level abstract features from the input and determine or generate at least one first image block feature; and at least one weight information is determined or generated based on the at least one first image block feature.

[0322] Optionally, at least one first image block feature can be converted into a vector of fixed length by global average pooling (GAP), and at least one weight information can be generated by a fully connected layer or other methods. Optionally, weight information can be learned from at least one first image block feature by an adaptive weighting mechanism using an additional neural network module to determine or obtain at least one weight information. Optionally, a specially designed attention module can be introduced through an attention mechanism to dynamically assign different weights to different spatial positions or channels according to at least one first image block feature to determine or obtain at least one weight information.

[0323] Optionally, at least one second image block feature corresponding to pixels of at least one reference area of ​​the current block is fused according to the neural network model to determine or generate a fused feature; at least one weight information is determined or generated according to the fused feature. Optionally, the second image block feature may include features determined or generated by full connection processing and / or convolution processing. Feature fusion further enriches feature representation, improves the prediction accuracy of weight information, and thus improves the prediction effect of inter-frame prediction.

[0324] For example, refer to Fig.19 , the neural network model based on the fully connected layer includes: an input layer, multiple hidden layers (i.e., fully connected layers) and an output layer, and the fully connected layer includes: a linear layer (Linear layer) and an activation function layer (e.g., a LeakyReLU layer). The pixels of the reference area of ​​the current block are input into the neural network model based on the fully connected layer, and the reference area includes: a reference area A composed of the upper adjacent pixels of the current block, a reference area B composed of the left adjacent pixels of the current block, and a reference area C composed of the upper left adjacent pixels of the current block. Since the pixels of the reference area may exist in the form of a two-dimensional or three-dimensional matrix, in order to be able to send the above data to the fully connected layer for processing, the input layer can be flattened to flatten the pixels of at least one reference area into a one-dimensional vector f, and the ReLU activation function is used to perform a nonlinear transformation to determine or obtain f i ; This operation introduces nonlinearity, allowing the model to learn more complex patterns. i It will enter the network composed of fully connected layers, each layer performs linear transformation and activation function processing, and outputs a new feature vector f r , to extract higher-level abstract features from the input. Then the high-level features f r With the low-level feature f i Concatenate on the channel dimension to generate the feature vector f c ; The feature vector f c It is fed into the linear layer and outputs at least one weight information through the output layer.

[0325] Optionally, the first model is determined or obtained according to the width and height of at least one reference area of ​​the current block. Exemplarily, when the width and height of the reference area are both less than or equal to X1 (for example, 8), and / or the output weight information is a weight matrix, and the output size height and width are both less than or equal to X1 (for example, 8), the first model is a neural network model based on a fully connected layer, and the hidden layer structure of the first model is shown in Table 10 below:

[0326] Table 10

[0327] Number of layers type parameter First floor Linear+LeakyReLU 320*2560 Second floor Linear+LeakyReLU 2560*2560 Third floor Linear+LeakyReLU 2560*2560 Fourth floor Linear+LeakyReLU 2560*2560 Fifth floor Linear 2880*(h*w)

[0328] Exemplarily, when at least one of the width and height of the reference area is greater than X1 (for example, 8), and / or the output weight information is a weight matrix, and at least one of the output size height and width is greater than X1 (for example, 8), the first model is a neural network model based on a fully connected layer, and the hidden layer structure of the first model is shown in Table 11 below:

[0329] Table 11

[0330]

[0331]

[0332] In this embodiment, when the pixels of the reference area are reconstructed reference pixels, since the reconstructed reference pixels provide information already existing in the frame, the requirements for extraction efficiency are relatively low, so a fully connected layer can be used to perform more refined feature extraction, improve the accuracy of weight information prediction, and the accuracy of the prediction block determined or generated based on the weight information. At the same time, the fully connected layer has higher flexibility and can process pixels of reference areas of different sizes (such as different numbers of layers), improve the flexibility of weight information and the flexibility of video image prediction, and thus improve the prediction effect during video encoding and / or decoding.

[0333] Optionally, a convolutional neural network, i.e., a convolutional neural network (CNNs), includes: an input layer, at least one hidden layer, and an output layer, wherein the hidden layer includes: a convolutional layer for extracting features; each convolutional layer may be followed by a pooling layer for reducing the spatial size and reducing the computational complexity; each convolutional layer may also be followed by a nonlinear activation function (e.g., ReLU) to introduce nonlinear factors so that the network can learn more complex patterns. As the depth of the convolutional neural network increases, the convolutional neural network can automatically learn feature representations from low-level to high-level, improving prediction accuracy.

[0334] Optionally, at least one third image block feature is determined or generated by performing convolution processing on pixels of at least one reference area of ​​the current block according to at least one convolution layer of the neural network model; and at least one weight information is determined or generated based on the at least one third image block feature.

[0335] Optionally, at least one third image block feature can be converted into a vector of fixed length by global average pooling (GAP), and at least one weight information can be generated by a fully connected layer or other methods. Optionally, weight information can be learned from at least one third image block feature by an adaptive weighting mechanism using an additional neural network module to determine or obtain at least one weight information. Optionally, a specially designed attention module can be introduced through an attention mechanism to dynamically assign different weights to different spatial positions or channels according to at least one third image block feature to determine or obtain at least one weight information.

[0336] Optionally, at least one fourth image block feature is determined or generated by performing convolution processing on the fused features corresponding to the pixels of at least one reference area of ​​the current block according to the neural network model; at least one weight information is determined or generated according to the at least one fourth image block feature. Since the convolution layer can learn the complex patterns of the input data and improve the ability of feature representation, the spatial relationship between adjacent pixels in the reference area can be captured by performing convolution processing on the fused features corresponding to the pixels of at least one reference area of ​​the current block, and the information from different sources is more effectively combined, thereby improving the overall performance and prediction effect.

[0337] For example, refer to Fig. 20 The convolutional neural network includes: an input layer, multiple hidden layers and an output layer. The hidden layers include: a convolutional layer, an activation function layer (e.g., a LeakyReLU layer) and a linear layer. The pixels of the reference area of ​​the current block are input into the convolutional neural network. The reference area includes: a reference area A composed of the upper adjacent pixels of the current block, a reference area B composed of the left adjacent pixels of the current block, and a reference area C composed of the upper left adjacent pixels of the current block. The pixels of at least one reference area are convolved through at least one convolutional layer, and nonlinearly transformed through the ReLU activation function after each convolutional layer to obtain the feature f. 1 、f 2 and f 3 . The feature f 1 、f 2 and f 3 The fusion splicing is performed on the channel dimension to determine or generate a new merged feature f containing three features to retain information from different convolution paths, allowing the model to combine low-level and high-level features and enhance representation capabilities. The fused feature f is processed again through the ReLU activation function to obtain f i ; This additional activation step helps to further improve the expressiveness and flexibility of the model. i Send it to the linear layer to get at least one weight information.

[0338] Optionally, the first model is determined or obtained according to the width and height of at least one reference area of ​​the current block. Exemplarily, when the width and height of the reference area are both less than or equal to X1 (e.g., 8), and / or the output weight information is a weight matrix, and the output size height and width are both less than or equal to X1 (e.g., 8), the first model is a convolutional neural network, and the hidden layer structure of the first model is shown in Table 12 below:

[0339] Table 12

[0340] Number of layers type parameter First floor Conv2d+LeakyReLU 3*3*1*8 Second floor Conv2d+LeakyReLU 3*3*8*8 Third floor Conv2d+LeakyReLU 3*3*8*8 Fourth floor Conv2d+LeakyReLU 3*3*8*8 Fifth floor Linear 2880*(h*w)

[0341] Exemplarily, when at least one of the width and height of the reference area is greater than X1 (e.g., 8), and / or the output weight information is a weight matrix, and at least one of the output height and width is greater than X1 (e.g., 8), the first model is a convolutional neural network, and the hidden layer structure of the first model is shown in Table 13 below:

[0342] Table 13

[0343] Number of layers type parameter First floor Conv2d+LeakyReLU 3*3*1*8 Second floor Conv2d+LeakyReLU 3*3*8*8 Third floor Conv2d+LeakyReLU 3*3*8*8 Fourth floor Conv2d+LeakyReLU 3*3*8*8 Fifth floor Conv2d+LeakyReLU 3*3*8*8 Sixth floor Conv2d+LeakyReLU 3*3*8*8 Seventh floor Linear 2880*(h*w)

[0344] In this embodiment, since the connection of the convolution layer is local and sparse, the number of parameters that the convolution neural network needs to learn is greatly reduced compared to other neural network models; at the same time, the convolution process is that the same convolution kernel slides over the entire input space and applies the same weights to detect specific types of features (such as edges, textures, etc.), so the number of parameters that the convolution neural network needs to learn can be further reduced, and the model can be more effectively trained with limited data, improve the learning efficiency of the model, and help prevent overfitting. And / or, by limiting each neuron to only connect to a local area of ​​the input, the convolution layer can focus on capturing local patterns and structures in the input data, and important information in the image often exists in local areas. Therefore, the accuracy of weight information prediction and the accuracy of prediction blocks determined or generated based on weight information can be improved through the convolution neural network, thereby improving the prediction effect in the video encoding and / or decoding process.

[0345] Optionally, the neural network model based on hybrid convolution and fully connected layers is an architecture that combines the advantages of convolutional neural networks (CNNs) and multi-layer perceptrons (MLPs), including a convolution layer part and a fully connected layer part. The spatial hierarchical structure of the input data, such as features such as edges, textures, and shapes in an image, can be automatically learned through the convolution layer. The convolution operation can effectively capture local patterns and reduce model complexity through a parameter sharing mechanism. The features extracted by the convolution layer can be flattened into a one-dimensional vector through the fully connected layer, and then fed into a series of fully connected layers. These layers are responsible for combining low-level features to determine or generate higher-level abstract representations. The neural network model based on hybrid convolution and fully connected layers can combine the advantages of two different types of layers, and can not only efficiently extract features from the input data, but also make accurate predictions based on the features. Optionally, the neural network model based on hybrid convolution and fully connected layers includes: an input layer, at least one convolution layer, at least one fully connected layer, and an output layer.

[0346] For example, refer to Fig.11 The neural network model based on hybrid convolution and fully connected layers includes: an input layer, multiple hidden layers and an output layer. The hidden layers include: a convolutional layer, a fully connected layer (i.e., a linear layer) and an activation function layer (e.g., a LeakyReLU layer). The pixels of the reference area of ​​the current block are input into the neural network model based on hybrid convolution and fully connected layers. The reference area includes: a reference area A composed of the upper adjacent pixels of the current block, a reference area B composed of the left adjacent pixels of the current block, and a reference area C composed of the upper left adjacent pixels of the current block. The pixels of at least one reference area are convolved through at least one convolution layer, and nonlinearly transformed through the ReLU activation function after each convolution layer to obtain the feature f. 1 、f 2 and f 3 . The feature f 1 、f 2 and f 3 Perform fusion and splicing in the channel dimension to determine or generate a new merged feature f containing three features 4 , to retain information from different convolution paths, allowing the model to combine low-level and high-level features and enhance representation capabilities. Multiple stacked linear layers are used to obtain the feature f 5 ; The feature f 4 and f 5 Concatenate in the channel dimension to form the fusion feature f i . Through at least one convolutional layer, the fusion feature f i A convolution process is performed to obtain at least one weight information.

[0347] Optionally, the first model is determined or obtained according to the width and height of at least one reference area of ​​the current block. Exemplarily, when the width and height of the reference area are both less than or equal to X1 (e.g., 8), and / or the output weight information is a weight matrix, and the output size height and width are both less than or equal to X1 (e.g., 8), the first model is a neural network model based on a hybrid convolution and fully connected layer, and the hidden layer structure of the first model is shown in Table 14 below:

[0348] Table 14

[0349] Number of layers type parameter The first convolutional layer Convolutional(3*3) 3*3*1*8 The second convolutional layer Convolutional(3*3) 3*3*8*8 First fully connected layer Linear+LeakyReLU 320*2560 Second fully connected layer Linear+LeakyReLU 2560*2560 The third convolutional layer Convolutional(3*3) 3*3*8*8

[0350] Exemplarily, when at least one of the width and height of the reference area is greater than X1 (e.g., 8), and / or the output weight information is a weight matrix, and at least one of the output size height and width is greater than X1 (e.g., 8), the first model is a neural network model based on a hybrid convolution and fully connected layer, and the hidden layer structure of the first model is shown in Table 15 below:

[0351] Table 15

[0352] Number of layers type parameter The first convolutional layer Convolutional(3*3) 3*3*1*8 The second convolutional layer Convolutional(3*3) 3*3*8*8 The third convolutional layer Convolutional(3*3) 3*3*8*8 The fourth convolutional layer Convolutional(3*3) 3*3*8*8 First fully connected layer Linear+LeakyReLU 2560*2560 Second fully connected layer Linear+LeakyReLU 2560*2560 The fifth convolutional layer Convolutional(3*3) 3*3*8*8 The sixth convolutional layer Convolutional(3*3) 3*3*8*8

[0353] In this embodiment, the neural network model based on the hybrid convolution and fully connected layer can combine the advantages of two different types of layer structures, which can not only efficiently extract features from the input data, but also make more accurate predictions based on the features.

[0354] Optionally, first, obtain N data sets. After obtaining the N data sets, divide the N data sets into two parts, one for training data sets and the other for testing data sets. Optionally, the number of training data sets is N1, the number of testing data sets is N2, and N1+N2=N. Next, train the neural network with the training data sets, and test the trained neural network with the testing data sets.

[0355] Optionally, each of the N1 training data sets is subjected to pixel value offset correction, and the reference pixels of the reference area input to the neural network are obtained according to the position information of each data set. After that, the obtained reference pixels are input to the neural network to train the neural network. After the neural network training is completed, each of the N test data sets is used to test the trained neural network to determine whether the trained neural network can make accurate predictions.

[0356] Optionally, if a matching neural network is trained for different image block sizes, the image block size of each of the N1 training data sets mentioned above is determined, and according to the value of the image block size, the reference pixels of the reference area corresponding to the training data set are input into the matching neural network. After that, the neural network is trained. Similarly, in the testing phase, the test data set corresponding to the gallery block size is also input into the matching neural network for testing.

[0357] Optionally, a plurality of video sequences for constructing a data set are first determined. Then, a data extraction module is set in the encoder and the decoder to extract data from the above video sequence. The extracted data includes: a frame index (poc), position information (x, y) of an image block (e.g., a coding unit), a size (width, height) of the image block, a channel type (compidx), a split angle (splitdir), a pixel value offset (shiftWeighted) and a compensation amount (offsetWeighted), an original image pixel value (org) of the image block, and at least one of a pixel value of a reference block of the image block.

[0358] Optionally, data is extracted from the above-mentioned multiple video sequences by an encoder and a decoder provided with a data extraction module. The original data is encoded using four QP values ​​(22, 27, 32, 37) to determine or generate corresponding bitstream files. The bitstream files are decoded using a decoder integrated with the data extraction module to determine or generate decoded reconstructed data.

[0359] Optionally, by Determine or generate a target label predicted by weight information to guide the training process of the neural network model, and then determine or obtain training data. i,j The target label predicted by the weight information, org i,j is the original pixel value of the current block, src0 i,j is the pixel value of the first reference block, src1 i,j is the pixel value of the second reference block.

[0360] Optionally, load data from a pre-built training dataset. The dataset contains the location, size, channel information, division angle, offset, original pixel value, and pixel value of the reference block of the current block. Normalize the loaded data. Take the pixels of the reference area of ​​the current block as input and the weight information as output. For example, if the input reference area includes reference area A, reference area B, and reference area C, the input sizes are recorded as [B, C 1 ,H 1 ,W 1 ]、[B,C 2 ,H2 ,W 2 ] and [B,C 3 ,H 3 ,W 3 ].

[0361] Optionally, in the case where the neural network model includes a fully connected layer, since the input data of the fully connected layer needs to be a two-dimensional structure of [B, C], the pixels of the reference area A, the reference area B, and the reference area C are flattened to determine or obtain the structural features of [B, C×H×W], and the flattened features are concatenated in the second dimension (i.e., C dimension) to obtain a [B, C 1 ×H 1 ×W 1 +C 2 ×H 2 ×W 2 +C 3 ×H 3 ×W 3 ] characteristics.

[0362] Optionally, in the case where the neural network model is a neural network model based on a hybrid convolutional and fully connected layer, the pixels of the reference region and the pixels of the reference block can be used as input, and the predicted pixels can be used as output. The input size is denoted as the first reference region [B,C 1 ,H 1 ,W 1 ], the second reference area [B,C 2 ,H 2 ,W 2 ], the third reference area [B,C 3 ,H 3 ,W 3 ], the first reference block [B,C 4 ,H 4 ,W 4 ], the second reference block [B,C 5 ,H 5 ,W 5 ]. For the features corresponding to the first reference block and the second reference block, the features can be extracted by convolution processing to obtain the corresponding [B,C 6 ,H 6 ,W 6 ]. In order to better integrate the obtained features with the features of the reference area, the features of each reference area can be expanded into the shape of [B, C, H, W] and then integrated.

[0363] Optionally, for the size where the width w of the current block is not equal to the height h, both w×h and h×w data in the data set are included in the training, and the h×w data can be transposed to obtain w×h. For the size where the width w is equal to the height h, in addition to using the original data for training, the data after the original data itself is transposed can be added as training data. Optionally, in the actual encoding and decoding process, since some positions may not have reconstructed pixels, that is, the encoder may not be able to obtain complete reference area information, resulting in missing pixel values. In order to simulate this situation and improve the generalization ability of the model, the data can be truncated and the pixel filling process in the video codec can be simulated to improve data diversity and adapt to the situation where complete reference template information cannot be obtained.

[0364] Optionally, set the loss function for model training; adopt a learning rate scheduling strategy to decay the learning rate every preset rounds to promote the convergence of the model. For example, the loss function L SmoothL1 (x) is:

[0365]

[0366] Optionally, the input data in the training data is input into the neural network model, and the prediction result is obtained by forward propagation. The prediction result is compared with the target label to determine or obtain the training loss. The gradient is calculated by the back propagation algorithm, and the model parameters are updated using the optimizer to minimize the loss function. After each preset number of iterations, the model is switched to the evaluation mode, and the validation set is forward propagated to calculate the validation loss. By comparing the training loss and the validation loss, the generalization ability of the model is monitored to prevent overfitting or underfitting. After the training is completed, a trained neural network model is obtained.

[0367] Optionally, after the neural network model is trained so that the performance of the neural network is optimized, the neural network model can be used to perform inter-frame prediction processing in the inter-frame prediction stage of the encoding and decoding process. For example, at least one weight information is determined or obtained based on pixels of at least one reference area of ​​the current block and the neural network model; a prediction block of the current block is determined or generated based on the at least one weight information and at least one reference block of the current block.

[0368] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0369] Optionally, in order to reduce the time complexity caused by the neural network model inference, the neural network model can be optimized for the video codec. For example, through model quantization, the floating-point parameters and calculations in the neural network model are converted into integer form to reduce the storage requirements of the model and accelerate the inference process. For example, the trained model is statically quantized to convert the trained floating-point weights and the output values ​​of the activation function into 8-bit integers to ensure that the model uses integer operations completely during inference. It not only significantly improves the inference speed of the model, but also greatly reduces memory usage, which is particularly suitable for embedded devices and mobile applications.

[0370] Optionally, a Small Adhoc Deep-Learning Library (SADL) dedicated to deep learning that supports efficient integer operations can be selected to ensure that on the basis of the quantized model, the reasoning process can make full use of hardware acceleration features, such as single instruction multiple data (SIMD) instruction set and multi-threaded parallel computing. Optionally, unnecessary functional modules and redundant codes in the SADL library can be eliminated by analyzing actual needs, and only the parts directly related to the neural network model reasoning in the embodiment of the present application are retained. This not only reduces the size of the library, reduces memory usage, but also reduces the overhead of function calls, further improving the reasoning efficiency. Optionally, customized optimization can also be performed for the specific structure of the model; for example, the calculation process of the fully connected layer and the convolutional layer is optimized, the number of memory accesses is reduced, and the cache hit rate is improved, thereby speeding up the reasoning speed.

[0371] Optionally, in order to speed up the reasoning process, a neural network model structure that is as simple as possible can be set. Under the premise of ensuring the prediction accuracy of the model, the computational complexity and parameter quantity of the model can be reduced by reducing the number of network layers, reducing the number of neurons or channels in each layer, and adopting a lightweight network architecture. And / or, pruning technology can be used to remove neurons or channels that have little impact on the final output to further reduce the size of the model. And / or, the number of parameters of the overall model can be reduced while maintaining the necessary expressive power through the parameter sharing and reuse mechanism of the convolutional layer. Optimizing the selection of activation functions and avoiding the use of activation functions with high computational complexity can also effectively accelerate the nonlinear transformation process.

[0372] Optionally, the processing method of the present application can be integrated into the inter-frame prediction module of the encoder and / or decoder; for example, an inter-frame prediction mode based on weight information is defined in the inter-frame prediction module as a class in the GPM model, which is responsible for managing and executing the reasoning process of the neural network model, including: model initialization, reference data extraction, reasoning process, weight information output and other functions. Optionally, the inter-frame prediction module is a module shared by the encoder and decoder, including: motion search, motion compensation and other modules.

[0373] Optionally, when the integrated encoder and / or decoder is started, a neural network model suitable for different image block sizes can be loaded first. For example, a set of different model sizes are predefined to accommodate different current block and prediction block size requirements. For each predefined size, a corresponding model file path is constructed, and then the model file is read from the specified path and loaded into memory. If the model file cannot be read, the system will log an error and terminate execution. After successful loading, the model is stored in the internal model list of the class for subsequent reasoning.

[0374] Optionally, before performing inter-frame prediction, at least one reference area of ​​the current block is determined or obtained for inference by the neural network model. Optionally, the width and height of the current block can be determined, and the corresponding total length of the reference area is selected according to the width and height. Next, the pixels of the reference area are collected. For example, including: the upper adjacent pixels, the left adjacent pixels and the upper left adjacent pixels of the current block. The upper adjacent pixels can collect the matrix formed by pixel data from the top boundary of the current block from bottom to top and from left to right; the left adjacent pixels can collect the matrix formed by pixel data from the left boundary of the current block from right to left and from top to bottom; the upper left adjacent pixels can collect the matrix formed by pixels from the top boundary of the current block from bottom to top and from right to left. If some positions lack valid pixel data, they can be filled with adjacent valid pixels. Optionally, the collected pixels of the reference area are normalized according to the bit depth to meet the input requirements of the neural network model. The normalized pixels of the reference area are packaged into input data sets respectively to prepare for the subsequent inference process.

[0375] Optionally, when the input data is ready, the neural network model can be used for reasoning to determine or generate weight information. Optionally, the corresponding neural network model is selected according to at least one of the width, height, block size and block area of ​​the current block. The input data is input into the selected model to ensure that the amount and size of the input data are consistent with the model expectations. If the input data also includes other supplementary data, such as reference blocks, etc., it can be directly obtained during the use of the GPM tool of the inter-frame prediction module.

[0376] Optionally, pass input data to the model for initialization to ensure that the model is in a state for reasoning. Call the model's reasoning function to process the input data (i.e., pixels in the reference area) and generate output results (i.e., weight information). If there is a problem during reasoning, return an error message. Extract the weight information from the model's reasoning results and store it in the weight buffer for subsequent use.

[0377] Optionally, the size of the weight buffer is adjusted according to at least one of the width, height, block size and block area of ​​the current block to accommodate all the generated weight information. The weight information output by inference is assigned to the weight buffer one by one to ensure that each pixel block of the current block has a corresponding weight value.

[0378] Optionally, in the encoder, the integration of the processing method of the embodiment of the present application is mainly concentrated on the transmission part of mode discrimination and signaling. The prediction mode discrimination of the encoder can select whether to adopt the inter-frame prediction mode based on weight information of the embodiment of the present application according to the principle of minimizing the rate distortion cost. The mode selected by the encoder can be transmitted through signaling to inform the decoder whether it is necessary to use the inter-frame prediction mode based on weight information of the embodiment of the present application for decoding.

[0379] Optionally, in the encoder, a geometric partitioning candidate list (i.e., a GPM candidate list) can be generated and the rate-distortion cost can be estimated. For example, a GPM candidate list is created to store the prediction results of different candidate modes. By calling the motion estimation module, the encoder obtains the GPM candidate modes available for the current block, which contain different motion vector combinations. For each candidate mode, the sum of the absolute differences of the corresponding prediction errors can be estimated using conventional methods and preliminarily sorted.

[0380] Optionally, in the encoder, it can be checked whether the current block has an available reference area to determine whether the inter-frame prediction mode based on weight information of the embodiment of the present application can be applied. If the reference area is available, the encoder can use a pre-trained neural network model to predict the weight information of the current block. For example, the encoder inputs the pixels in the reference area consisting of the upper adjacent pixels, the left adjacent pixels and the upper left adjacent pixels of the current block into the neural network to determine or generate weight information. The weight information obtained based on the prediction and the reference block of the current block are weighted and fused to generate a prediction block. The error between the prediction block and the original image (i.e., the current block) is calculated. The prediction error is combined with its bit rate overhead to calculate its overall rate distortion cost.

[0381] Optionally, in the encoder, all candidate GPM modes can be sorted according to their estimated rate-distortion costs, from low to high. Optionally, an early stopping strategy can be applied, that is, if the cost of some candidate modes is significantly higher than the current best mode, further evaluation of these modes is skipped to improve coding efficiency. The encoder can select the geometric partitioning mode with the lowest rate-distortion cost as the final coding mode. If the inter-frame prediction mode based on weight information of an embodiment of the present application shows a lower cost, this mode is preferred.

[0382] Optionally, the encoder can determine or obtain the residual between the prediction block and the current block, and transform, quantize and entropy encode the residual to achieve effective data compression. The encoder transmits the selected mode information through the code stream; for example, if the mode corresponding to the processing method of the embodiment of the present application is selected, the signaling corresponding to the inter-frame prediction mode based on weight information of the embodiment of the present application is transmitted, and the motion vector combination candidate index is not required to transmit the original division method index of the GPM, so as to save the signaling required to be transmitted for the division method.

[0383] Optionally, in the decoder, the integration of the processing method of the embodiment of the present application is mainly focused on the reading of signaling and the application of the mode. The decoder can first parse the signaling information from the code stream, and decide which motion compensation mode to adopt based on the parsed mode flag. If the flag corresponding to the inter-frame prediction mode based on weight information of the present application is set, the decoder calls the code of the inter-frame prediction module integrated into the inter-frame prediction mode based on weight information of the present application, and uses the weight information predicted by the neural network model to perform weighted fusion on the reference block to reconstruct the final image block (ie, the prediction block).

[0384] Optionally, the inter-frame prediction performance of the processing method in the video codec can be evaluated according to the Common Test Conditions (CTC). The test process strictly follows the test sequence, quantization parameter QP, encoding profile and evaluation indicators defined in CTC to ensure the reliability and repeatability of the experimental results. For example, the test process includes:

[0385] Step 1: Coding configuration. According to the definition in CTC, select the test sequence suitable for random access (RA) and low delay (LD) conditions. For each test sequence, under each test condition, use the coding profile defined in CTC to encode to ensure the consistency of the encoding process.

[0386] Step 2: Encode all test sequences using four QP values ​​(22, 27, 32, 37) using an encoder that integrates the processing method of the present application, determine or generate corresponding bitstream files, and decode the generated bitstream files using a decoder of the same version to restore and reconstruct the video sequence.

[0387] Step 3, performance evaluation. Extract test results from the encoding and decoding logs, including bit rate, peak signal-to-noise ratio, and encoding and decoding time. Calculate rate-distortion performance based on the bit rate and peak signal-to-noise ratio results at different QPs to evaluate the bit rate savings of this method compared to the benchmark encoder and / or decoder.

[0388] Method ten: pixels and regression model of at least one reference area of ​​the current block.

[0389] Optionally, pixels of at least one reference area of ​​the current block may be used as input of the regression model, and the relationship between pixels of the reference area of ​​the current block and weight information learned by the regression model may be used to output at least one weight information.

[0390] Optionally, at least one weight information is determined or obtained based on pixels of at least one reference area of ​​the current block and a regression model; and a prediction block of the current block is determined or generated based on the at least one weight information and at least one reference block of the current block.

[0391] Alternatively, a regression model is a statistical method used to establish the relationship between independent variables (features) and dependent variables (targets). A regression model can construct a mathematical function by analyzing the relationship between known data points so that the function can predict the value of new unknown data points as accurately as possible.

[0392] Optionally, the regression model includes at least one of a linear regression model, a polynomial regression model, a logistic regression model, a support regression model, a decision tree regression model, a random forest regression model, and a gradient boosting regression model.

[0393] Now, let's use the linear regression model to determine the weight information. Assume that the current block includes two reference blocks P1 and P2, the reference area of ​​the current block is R, the weight information is the weight coefficients W1 and W2, and each pixel Py in the prediction block is represented by formula (1). Formula (1) is a linear model, as shown below:

[0394] Py=W1×P1[i,j]+W2×P2[i,j]+c;

[0395] Optionally, c is a bias term. Optionally, c is equal to 0.

[0396] First, the block vector or motion vector associated with the current block is used in the reference region R to determine the reference regions R1 and R2 corresponding to the reference region R. Next, the pixels in the reference region R are taken as Py, and the pixels at corresponding positions in the reference region R1 and the reference region R2 are taken as P1[i, j] and P2[i, j], respectively. The values ​​of the weight coefficients W1 and W2 in formula (1) are solved by solving the mean square error minimum (MSEminimization).

[0397] Optionally, when the weight information is a weight vector or a weight matrix, the value of the corresponding weight vector or weight matrix can also be solved by the above method.

[0398] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0399] In this embodiment, by using the regression model, the pixels of the reference area of ​​the current block are used to directly predict the weight of the current block, which can avoid the influence of conventional linear division on the prediction accuracy, and take into account the correlation between the current block and its reference area, thereby improving the prediction effect of inter-frame prediction. And / or, it can also effectively save the signaling required for the transmission of the division method and reduce the signaling overhead. And / or, the computational complexity of the regression model is low, and the training and prediction process can be completed quickly.

[0400] Method eleven: pixels of at least one reference area of ​​the current block and a matrix-based mathematical model.

[0401] Optionally, the matrix-based mathematical model may be a mathematical model constructed by obtaining a matrix by simplifying a neural network model.

[0402] Optionally, pixels of at least one reference area of ​​the current block may be used as input to a matrix-based mathematical model, and at least one weight information may be outputted using the relationship between pixels of the reference area of ​​the current block and weight information learned by the matrix-based mathematical model.

[0403] Optionally, at least one weight information is determined or obtained based on pixels of at least one reference area of ​​the current block and a matrix-based mathematical model; and a prediction block of the current block is determined or generated based on the at least one weight information and at least one reference block of the current block.

[0404] Optionally, a mathematical model is a type of model described in mathematical language. A mathematical model can be one or a group of algebraic equations, differential equations, difference equations, integral equations or statistical equations, or some appropriate combination thereof, through which the interrelationships or causal relationships between the variables of the system are quantitatively or qualitatively described. In addition to mathematical models described by equations, there are also models described by other mathematical tools, such as algebra, geometry, topology, mathematical logic, etc. It can be understood that mathematical models describe the behavior and characteristics of the system rather than the actual structure of the system.

[0405] Optionally, a matrix-based mathematical model refers to a type of model that uses matrix operations and linear algebra principles to describe and solve problems. It represents data as matrices and uses matrix operations (such as multiplication, inverse, eigenvalue decomposition, etc.) to perform calculations and analysis. It can efficiently handle large-scale data sets and complex mathematical relationships.

[0406] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0407] In this embodiment, by using a matrix-based mathematical model, the pixels of the reference area of ​​the current block are used to directly predict the weight of the current block, which can avoid the influence of conventional linear division on the prediction accuracy, and take into account the correlation between the current block and its reference area, thereby improving the prediction effect of inter-frame prediction. And / or, it can also effectively save the signaling required for the division method transmission and reduce the signaling overhead. And / or, the matrix-based mathematical model can effectively represent multi-dimensional data, reduce complexity and improve interpretability.

[0408] Method 12: pixels of at least one reference area of ​​the current block and a mathematical model based on a lookup table.

[0409] Optionally, pixels of at least one reference area of ​​the current block may be used as input to a mathematical model based on a lookup table, and at least one weight information may be output using the relationship between pixels of the reference area of ​​the current block and weight information learned by the mathematical model based on the lookup table.

[0410] Optionally, at least one weight information is determined or obtained based on pixels of at least one reference area of ​​the current block and a mathematical model based on a lookup table; and a prediction block of the current block is determined or generated based on the at least one weight information and at least one reference block of the current block.

[0411] Alternatively, a mathematical model based on a lookup table is a technique that enables fast query by pre-calculating and storing a series of input-output mapping relationships. A lookup table is essentially a two-dimensional or multi-dimensional array, where each element corresponds to an output value under specific input conditions. In practical applications, when it is necessary to calculate the output corresponding to a certain input, the result can be directly read from the lookup table without having to re-calculate the complex calculation.

[0412] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0413] In this embodiment, by using a mathematical model based on a lookup table and directly performing weight prediction on the current block using pixels in the reference area of ​​the current block, it is possible to avoid the influence of conventional linear division on the prediction accuracy, and take into account the correlation between the current block and its reference area, thereby improving the prediction effect of inter-frame prediction. And / or, it is also possible to effectively save the signaling required for transmission in the division method and reduce the signaling overhead. And / or, the mathematical model based on the lookup table can significantly reduce the amount of calculation and reduce the processor burden at runtime by completing most of the calculation work in advance and storing the results in the lookup table; and the pre-calculated results can be retrieved at a very fast speed to improve the prediction rate of weight information, thereby improving the prediction efficiency in the video encoding and / or decoding process.

[0414] Method thirteen: pixels of at least one reference area of ​​the current block and a mathematical model based on a normalization operation.

[0415] Optionally, pixels of at least one reference area of ​​the current block may be used as input to a mathematical model based on normalization operation, and at least one weight information may be outputted using the relationship between pixels of the reference area of ​​the current block and weight information learned by the mathematical model based on normalization operation.

[0416] Optionally, at least one weight information is determined or obtained based on pixels of at least one reference area of ​​the current block and a mathematical model based on a normalization operation; and a prediction block of the current block is determined or generated based on the at least one weight information and at least one reference block of the current block.

[0417] Alternatively, a mathematical model based on normalization operation refers to a method that simplifies calculations, improves model performance, and enhances the interpretability of results by converting input data into a standardized range. Normalization operations are often used to process data of different scales to ensure that individual features are compared or combined on the same basis.

[0418] In this embodiment, the reference area is determined or obtained according to any one of the methods 1 to 6 in the second embodiment.

[0419] In this embodiment, by using a mathematical model based on normalization operation and directly performing weight prediction on the current block using pixels in the reference area of ​​the current block, the influence of conventional linear division on the prediction accuracy can be avoided, and the correlation between the current block and its reference area can be taken into account, thereby improving the prediction effect of inter-frame prediction. And / or, the signaling required for the division method transmission can be effectively saved, the signaling overhead can be reduced, and the compression efficiency of the video can be improved.

[0420] Fifth embodiment

[0421] The present application also provides a processing device, referring to Fig. 22 , the processing device comprises:

[0422] The processing module A10 is used to perform prediction processing on the current block according to at least one weight information, and determine or generate a prediction block.

[0423] Optionally, the weight information is determined or obtained from at least one reference area of ​​the current block.

[0424] Optionally, the processing module A10 is further configured to:

[0425] Determine or obtain at least one reference area according to at least one of an upper adjacent pixel, an upper non-adjacent pixel, a left adjacent pixel, a left non-adjacent pixel, an upper left adjacent pixel, and an upper left non-adjacent pixel of the current block;

[0426] Determine or obtain at least one reference area according to at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block, and a default block corresponding to the current block;

[0427] Determine or obtain at least one reference area according to at least one of the width, height, block size and block area of ​​the current block;

[0428] Determine or obtain at least one reference area according to the candidate block determined or obtained based on the candidate motion vector or the candidate block vector of the current block;

[0429] If the first information of the current block satisfies the first condition, the reference area is the first reference area;

[0430] If the first information of the current block does not satisfy the first condition, the reference area is the second reference area.

[0431] Optionally, the first information of the current block satisfies the first condition, including at least one of the following:

[0432] The value of the first information is a first value;

[0433] The value of the first information is within a first numerical range;

[0434] The value of the first information is greater than or equal to the first threshold;

[0435] The value of the first information is less than or equal to the second threshold.

[0436] Optionally, the processing module A10 includes:

[0437] A first determining unit, configured to determine or obtain at least one weight information according to pixels of at least one reference area of ​​the current block and the first model;

[0438] The second determining unit is used to determine or generate a prediction block of the current block according to at least one weight information and at least one reference block of the current block.

[0439] Optionally, the at least one weight information is determined or obtained according to at least one of the following:

[0440] Pixels and a neural network model of at least one reference region of the current block;

[0441] pixels and a regression model of at least one reference region of the current block;

[0442] Pixels of at least one reference region of the current block and a matrix-based mathematical model;

[0443] Pixels of at least one reference area of ​​the current block and a mathematical model based on a lookup table;

[0444] Pixels of at least one reference area of ​​the current block and a mathematical model based on a normalization operation.

[0445] Optionally, the first determining unit is further configured to:

[0446] Determine or obtain a first neural network model from a set of neural network models;

[0447] At least one weight information is determined or obtained based on pixels of at least one reference area of ​​the current block and the first neural network model.

[0448] Optionally, at least one reference block of the current block is determined or obtained according to an index obtained from a bitstream.

[0449] Optionally, the first model is determined or obtained according to at least one of the following:

[0450] at least one of the width, height, size and area of ​​at least one reference region of the current block;

[0451] At least one of the width, height, block size, and block area of ​​the current block.

[0452] The processing device provided in the embodiment of the present application has similar implementation principles and beneficial effects to the technical solutions shown in the above-mentioned corresponding method embodiments, and will not be described in detail here.

[0453] An embodiment of the present application also provides a processing device, including a memory and a processor. A processing program is stored in the memory, and when the processing program is executed by the processor, the steps of the processing method in any of the above embodiments are implemented.

[0454] An embodiment of the present application further provides a storage medium, on which a processing program is stored. When the processing program is executed by a processor, the steps of the processing method in any of the above embodiments are implemented.

[0455] In the embodiments of the processing device and storage medium provided in the present application, all technical features of any of the above-mentioned processing method embodiments may be included, and the expanded and explained contents of the specification are basically the same as those of the embodiments of the above-mentioned methods, and will not be repeated here.

[0456] The embodiment of the present application further provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes the methods in the above various possible implementation modes.

[0457] An embodiment of the present application also provides a chip, including a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that a device equipped with the chip executes the methods in various possible implementation modes as described above.

[0458] It is understood that the above scenarios are only examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of the present application. The technical solutions of the present application can also be applied to other scenarios. For example, it is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0459] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0460] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0461] The units in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0462] In the present application, the same or similar terminology concepts, technical solutions and / or application scenario descriptions are generally described in detail only the first time they appear. When they appear again later, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of the present application, for the same or similar terminology concepts, technical solutions and / or application scenario descriptions that are not described in detail later, reference can be made to the previous related detailed descriptions.

[0463] In the present application, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0464] The various technical features of the technical solution of the present application can be arbitrarily combined. In order to make the description concise, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0465] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, a controlled terminal, or a network device, etc.) to execute the method of each embodiment of the present application.

[0466] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a storage medium, or transmitted from one storage medium to another storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. Available media can be magnetic media, (e.g., floppy disk, storage disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state storage disk Solid State Disk (SSD)), etc.

[0467] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A processing method, characterized in that: Includes steps: S10, performing prediction processing on the current block according to at least one weight information to determine or generate a prediction block.

2. The processing method according to claim 1, characterized in that The weight information is determined or obtained from at least one reference area of ​​the current block.

3. The processing method according to claim 2, characterized in that: The reference region is determined or obtained by at least one of the following methods: Determine or obtain at least one reference area according to at least one of an upper adjacent pixel, an upper non-adjacent pixel, a left adjacent pixel, a left non-adjacent pixel, an upper left adjacent pixel, and an upper left non-adjacent pixel of the current block; Determine or obtain at least one reference area according to at least one of a neighboring block, a non-neighboring block, a co-located block, a time domain block, and a default block corresponding to the current block; Determine or obtain at least one reference area according to at least one of the width, height, block size and block area of ​​the current block; Determine or obtain at least one reference area according to the candidate block determined or obtained based on the candidate motion vector or the candidate block vector of the current block; If the first information of the current block satisfies the first condition, the reference area is the first reference area; If the first information of the current block does not satisfy the first condition, the reference area is the second reference area.

4. The processing method according to claim 3, characterized in that: The first information of the current block satisfies the first condition, including at least one of the following: The value of the first information is a first value; The value of the first information is within a first numerical range; The value of the first information is greater than or equal to the first threshold; The value of the first information is less than or equal to the second threshold.

5. The processing method according to any one of claims 1 to 4, characterized in that: Step S10 includes: Determine or obtain at least one weight information according to pixels of at least one reference area of ​​the current block and the first model; A prediction block of the current block is determined or generated according to the at least one weight information and at least one reference block of the current block.

6. The processing method according to claim 5, characterized in that: The at least one weight information is determined or obtained according to at least one of the following: Pixels and a neural network model of at least one reference region of the current block; pixels and a regression model of at least one reference region of the current block; pixels of at least one reference region of the current block and a matrix-based mathematical model; Pixels of at least one reference area of ​​the current block and a mathematical model based on a lookup table; Pixels of at least one reference area of ​​the current block and a mathematical model based on a normalization operation.

7. The processing method according to claim 6, characterized in that: Determining or obtaining at least one weight information according to pixels of at least one reference area of ​​the current block and the neural network model, including: Determine or obtain a first neural network model from a set of neural network models; At least one weight information is determined or obtained based on pixels of at least one reference area of ​​the current block and the first neural network model.

8. The processing method according to claim 5, characterized in that: At least one reference block of the current block is determined or obtained according to an index obtained from a bitstream; and / or, the first model is determined or obtained according to at least one of the following: at least one of the width, height, size and area of ​​at least one reference region of the current block; At least one of the width, height, block size, and block area of ​​the current block.

9. A processing device, characterized in that: include: A memory and a processor, wherein a processing program is stored in the memory, and when the processing program is executed by the processor, the steps of the processing method according to any one of claims 1 to 8 are implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the processing method according to any one of claims 1 to 8.

Images