Image processing method, processing equipment and storage medium

Abstract

The invention provides an image processing method, processing equipment and a storage medium. The image processing method comprises the following steps: filtering at least one image block according to spatial domain features of the at least one image block; according to the technical scheme, the filtering effect of filtering processing can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, processing device and storage medium. Background Art

[0002] Existing high-efficiency video coding frameworks, such as Neural Network Based Video Coding (NNVC) and / or Enhanced Compression Model (ECM), propose a video frame encoding technology to improve coding performance without significantly increasing computational complexity.

[0003] During the process of conceiving and implementing this application, the inventors discovered at least the following problems: During the loop filtering stage of the encoding and decoding process, for example, a low complexity neural network loop filter (LC-NNLF) structure is introduced in NNVC for loop filtering. Due to the large number of image block features, the neural network needs to process multiple image block features and cannot focus on key image block features, resulting in poor processing performance and affecting the filtering effect, which needs to be improved.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] In response to the above technical problems, the present application provides an image processing method, a processing device and a storage medium, aiming to solve the technical problem of how to improve the filtering effect of filtering processing.

[0006] The present application provides an image processing method, which can be applied to a processing device, comprising the steps of:

[0007] S1 , performing filtering processing on at least one image block according to a spatial domain feature of the at least one image block.

[0008] Optionally, the spatial domain feature is determined or obtained according to at least one of the following:

[0009] Statistical characteristic values ​​of a pixel to be filtered and at least one neighboring pixel in at least one image block;

[0010] Statistical characteristic values ​​of at least two non-neighbor pixels in at least one image block;

[0011] a statistical feature value of a pixel to be filtered and at least one of the following: a pixel to be filtered, and at least one upper neighboring pixel of the pixel to be filtered, at least one left neighboring pixel of the pixel to be filtered, at least one right neighboring pixel of the pixel to be filtered, and at least one lower neighboring pixel of the pixel to be filtered;

[0012] The weighted statistical characteristic values ​​of the pixel to be filtered and the pixels adjacent to the pixel to be filtered in at least one image block;

[0013] The statistical characteristic value of at least one pixel in the outermost layer of a window centered on the pixel to be filtered, wherein the size parameter of the window is less than or equal to the size parameter of the image block;

[0014] The statistical eigenvalues ​​of eight pixels that are not adjacent to the pixel to be filtered;

[0015] A statistical characteristic value of a sub-block to be filtered in at least one image block containing at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel;

[0016] The at least one image block includes a statistical characteristic value of at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel in the area to be filtered.

[0017] Optionally, step S1 includes at least one of the following:

[0018] Determining or obtaining at least one spatial attention weight based on at least one spatial feature and at least one lookup table, and performing filtering processing on an intermediate element block corresponding to at least one image block based on the at least one spatial attention weight;

[0019] Determining or obtaining at least one spatial attention weight based on at least one spatial feature and at least one neural network, and performing filtering processing on an intermediate element block corresponding to at least one image block based on the at least one spatial attention weight;

[0020] At least one spatial attention weight is determined or obtained based on at least one spatial feature and at least one activation function, and filtering is performed on an intermediate element block corresponding to at least one image block based on the at least one spatial attention weight.

[0021] Optionally, at least one of the following is also included:

[0022] Determining or obtaining at least one intermediate element block based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block;

[0023] The lookup table structure includes at least one of the following:

[0024] at least one lookup table;

[0025] at least one neural network module;

[0026] At least two search branches, at least one search branch having the same input as another search branch;

[0027] at least two search branches, the input of at least one search branch being determined or obtained based on the output of another search branch;

[0028] At least two search branches, the at least two search branches being located in at least one channel corresponding to at least one pixel to be filtered or a first filtered intermediate element;

[0029] At least two search branches, the search tables of the at least two search branches are searched in a parallel search and / or serial search manner;

[0030] at least two lookup tables, wherein the input of at least one lookup table is determined or obtained according to the type, size parameter and / or input range of the other lookup table;

[0031] At least two lookup tables, the at least two lookup tables being located in at least one channel corresponding to at least one pixel to be filtered or a first filtered intermediate element;

[0032] At least two lookup tables, wherein the lookup mode of the at least two lookup tables in the same channel is parallel search and / or serial search;

[0033] At least two lookup tables, wherein the lookup tables in at least two channels are searched in parallel and / or serially;

[0034] at least one search branch and at least one neural network module, wherein an input of the at least one neural network module is the same as an input of the at least one search branch;

[0035] at least one search branch and at least one neural network module, wherein an input of the at least one neural network module is determined or obtained based on an output of the at least one search branch;

[0036] At least one lookup table and at least one neural network module, wherein an input of the at least one neural network module is determined or obtained according to a type, size parameter and / or input range of the at least one lookup table;

[0037] at least two lookup tables, wherein the output ranges of the at least two lookup tables are the same and / or the output ranges of the lookup tables are different;

[0038] at least two lookup tables, at least two of which have the same input range and / or different input ranges;

[0039] At least one first residual module based on a lookup table, the first residual module comprising a first branch including at least one lookup table structure, an adder, and a second branch including a short-circuit structure;

[0040] At least one second residual module based on a lookup table, a first input element of the second residual module passes through a first branch containing at least one lookup table structure to obtain a fourth output element, and a first input element of the third residual module passes through a second branch containing a short-circuit structure to obtain a fifth output element that is the same as the first input element, and the fourth output element and the fifth output element are weightedly added by an adder to obtain a sixth output element.

[0041] Optionally, the image processing method further includes at least one of the following:

[0042] The spatial attention weights corresponding to at least two first intermediate elements in at least one intermediate element block are the same and / or the spatial attention weights corresponding to the first intermediate elements are different;

[0043] The spatial attention weight corresponding to at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to another first intermediate element;

[0044] The spatial attention weight corresponding to at least one intermediate element region containing at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to at least one intermediate element region containing at least one other first intermediate element;

[0045] The spatial attention weight corresponding to at least one intermediate element sub-block containing at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to at least one intermediate element sub-block containing at least one other first intermediate element.

[0046] Optionally, filtering the intermediate element block corresponding to the at least one image block according to the at least one spatial attention weight includes at least one of the following:

[0047] Determine or obtain at least one filter block or filtered intermediate element according to a result of performing spatial attention enhancement on a first intermediate element in at least one intermediate element block according to at least one spatial attention weight;

[0048] Determine or obtain at least one filter block or filtered intermediate element based on a result of performing spatial attention enhancement on at least one intermediate element region in at least one intermediate element block according to at least one spatial attention weight;

[0049] At least one filter block or filtered intermediate element is determined or obtained as a result of performing spatial attention enhancement on at least one intermediate element sub-block in at least one intermediate element block according to at least one spatial attention weight.

[0050] Optionally, the image processing method further includes: determining or obtaining spatial features of at least one image block after image preprocessing.

[0051] Optionally, the image preprocessing includes at least one of the following:

[0052] Image sharpening;

[0053] Gradient calculation;

[0054] Transformation;

[0055] Perform texture detection based on the detection operator.

[0056] The present application also provides a processing device, comprising:

[0057] The processing module is used to perform filtering processing on at least one image block according to the spatial domain characteristics of the at least one image block.

[0058] The present application also provides a processing device, comprising: a memory and a processor, wherein an image processing program is stored in the memory, and when the image processing program is executed by the processor, the steps of any of the above-mentioned image processing methods are implemented.

[0059] The present application also provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-mentioned image processing methods.

[0060] As described above, the image processing method of the present application can be applied to a processing device, including: performing filtering processing on at least one image block based on the spatial characteristics of the at least one image block. Through the technical solution of the present application, when performing filtering processing on at least one image block, the spatial characteristics of the at least one image block are comprehensively considered to determine the key image block features that need to be processed in the image block, thereby focusing on the key image block features during the filtering processing, thereby improving the filtering effect of the filtering processing on the at least one image block. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for describing the embodiments. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without inventive work.

[0062] Figure 1 A schematic diagram of the hardware structure of a mobile terminal for implementing various embodiments of the present application;

[0063] Figure 2 A communication network system architecture diagram provided in an embodiment of the present application;

[0064] Figure 3 A schematic diagram of the hardware structure of a controller 140 provided in this application;

[0065] Figure 4 A schematic diagram of the hardware structure of a network node 150 provided in this application;

[0066] Figure 5 is a flowchart of an image processing method according to the first embodiment;

[0067] Figure 6 It is a flowchart of encoding and decoding in the image processing method;

[0068] Figure 7 It is a filtering flow chart in the image processing method;

[0069] Figure 8 It is a schematic diagram of spatial feature extraction of an image block in an image processing method;

[0070] Figure 9 It is a schematic diagram of a processing module of a processing device.

[0071] The purpose of this application, its features, and advantages will be further described in conjunction with the embodiments and with reference to the accompanying drawings. The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and the accompanying text are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of this application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0072] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0073] It should be noted that, in this document, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0074] It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to a determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms "comprising" and "including" indicate the presence of the described features, steps, operations, elements, components, items, types, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, types, and / or groups. The terms "or," "and / or," "including at least one of the following," etc., used herein, may be interpreted as inclusive, or mean any one or any combination. For example, “comprising at least one of the following: A, B, C” means “any of the following: A; B; C; A and B; A and C; B and C; A and B and C”; and for another example, “A, B or C” or “A, B and / or C” means “any of the following: A; B; C; A and B; A and C; B and C; A and B and C”. An exception to this definition will occur only when a combination of elements, functions, steps or operations are inherently mutually exclusive in some manner.

[0075] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and they can be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0076] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0077] It should be noted that in this article, step codes such as S1 are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence.

[0078] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0079] In the subsequent description, the use of suffixes such as "module", "component" or "unit" to represent elements is only for the purpose of facilitating the description of the present application and has no specific meaning. Therefore, "module", "component" or "unit" can be used interchangeably.

[0080] The processing device in this application may be a smart terminal or a server. Optionally, the smart terminal may be implemented in various forms. For example, the smart terminal described in this application may include smart terminals such as mobile phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, and fixed terminals such as digital TVs and desktop computers.

[0081] The subsequent description will be made by taking a mobile terminal as an example. It will be understood by those skilled in the art that, in addition to components specifically used for mobile purposes, the configuration according to the embodiments of the present application can also be applied to fixed-type terminals.

[0082] See also Figure 1 , which is a schematic diagram of the hardware structure of a mobile terminal for implementing various embodiments of the present application. The mobile terminal 100 may include: an RF (Radio Frequency) unit 101, a WiFi module 102, an audio output unit 103, an A / V (audio / video) input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, a processor 110, and a power supply 111. Those skilled in the art will understand that Figure 1 The structure of the mobile terminal shown in the figure does not constitute a limitation to the mobile terminal. The mobile terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0083] The following combination Figure 1 A detailed introduction to the various components of the mobile terminal:

[0084] The RF unit 101 can be used to send and receive information or receive signals during calls. Specifically, it receives downlink information from the base station and transmits it to the processor 110 for processing. It also transmits uplink data to the base station. Typically, the RF unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, and more. Furthermore, the RF unit 101 can communicate with the network and other devices via wireless communication. The above-mentioned wireless communications may use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution), TDD-LTE (Time Division Duplexing-Long Term Evolution), 5G and 6G, etc.

[0085] WiFi is a short-range wireless transmission technology. Mobile terminals can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 102. It provides users with wireless broadband Internet access. Figure 1 The WiFi module 102 is shown, but it is understandable that it is not an essential component of the mobile terminal and can be omitted as needed without changing the essence of the invention.

[0086] The audio output unit 103 can convert audio data received by the RF unit 101 or the WiFi module 102 or stored in the memory 109 into an audio signal and output it as sound when the mobile terminal 100 is in a call signal reception mode, a talk mode, a recording mode, a voice recognition mode, a broadcast reception mode, or the like. Furthermore, the audio output unit 103 can also provide audio output related to a specific function performed by the mobile terminal 100 (e.g., a call signal reception sound, a message reception sound, etc.). The audio output unit 103 may include a speaker, a buzzer, or the like.

[0087] The A / V input unit 104 is used to receive audio or video signals. The A / V input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos captured by an image capture device (e.g., a camera) in video capture mode or image capture mode. The processed image frames may be displayed on the display unit 106. The image frames processed by the GPU 1041 may be stored in the memory 109 (or other storage medium) or transmitted via the RF unit 101 or the WiFi module 102. The microphone 1042 may receive sound (audio data) in operating modes such as a phone call mode, a recording mode, and a voice recognition mode, and may process such sound into audio data. In the phone call mode, the processed audio (voice) data may be converted into a format that can be transmitted to a mobile communication base station via the RF unit 101. The microphone 1042 may implement various types of noise cancellation (or suppression) algorithms to eliminate (or suppress) noise or interference generated during the reception and transmission of audio signals.

[0088] The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Optionally, the light sensor includes an ambient light sensor and a proximity sensor. Optionally, the ambient light sensor can adjust the brightness of the display panel 1061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 1061 and / or the backlight when the mobile terminal 100 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0089] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0090] The user input unit 107 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile terminal. Optionally, the user input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any other suitable object or accessory on or near the touch panel 1071) and drive the corresponding connection device according to a pre-set program. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Optionally, the touch detection device detects the user's touch direction and detects the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 110. It can also receive commands sent by the processor 110 and execute them. In addition, the touch panel 1071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1071, the user input unit 107 may further include other input devices 1072. Optionally, the other input devices 1072 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power keys, etc.), a trackball, a mouse, a joystick, etc., and the specifics are not limited here.

[0091] Optionally, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the processor 110 to determine the type of touch event. The processor 110 then provides a corresponding visual output on the display panel 1061 according to the type of touch event. Figure 1 In the embodiment, the touch panel 1071 and the display panel 1061 are two independent components to realize the input and output functions of the mobile terminal. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize the input and output functions of the mobile terminal, which is not limited here.

[0092] The interface unit 108 serves as an interface through which at least one external device can be connected to the mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 108 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the mobile terminal 100 or may be used to transmit data between the mobile terminal 100 and an external device.

[0093] Memory 109 can be used to store software programs and various data. Memory 109 may primarily include a program storage area and a data storage area. Optionally, the program storage area may store an operating system and at least one application required for a function (such as a sound playback function or an image playback function); the data storage area may store data generated based on the use of the mobile phone (such as audio data, a phone book, etc.). Furthermore, memory 109 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0094] Processor 110 is the control center of the mobile terminal, connecting all components of the mobile terminal using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 109 and accessing data stored in memory 109, it executes various functions of the mobile terminal and processes data, thereby providing overall monitoring of the mobile terminal. Processor 110 may include one or more processing units; preferably, processor 110 may integrate an application processor and a modem processor. Optionally, the application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 110.

[0095] The mobile terminal 100 may also include a power supply 111 (such as a battery) for supplying power to various components. Preferably, the power supply 111 may be logically connected to the processor 110 through a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0096] although Figure 1 Not shown, the mobile terminal 100 may further include a Bluetooth module, etc., which will not be described in detail here.

[0097] To facilitate understanding of the embodiments of the present application, the communication network system on which the mobile terminal of the present application is based is described below.

[0098] See also Figure 2 , Figure 2 A communication network system architecture diagram is provided for an embodiment of the present application. The communication network system is an LTE system of universal mobile communication technology. The LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203 and an operator's IP service 204, which are connected in sequence.

[0099] Optionally, UE201 may be the above-mentioned terminal 100, which will not be described in detail here.

[0100] E-UTRAN 202 includes eNodeB 2021 and other eNodeBs 2022 . Optionally, eNodeB 2021 may be connected to other eNodeBs 2022 via a backhaul (eg, an X2 interface). eNodeB 2021 is connected to EPC 203 , and eNodeB 2021 may provide access from UE 201 to EPC 203 .

[0101] EPC 203 may include an MME (Mobility Management Entity) 2031, an HSS (Home Subscriber Server) 2032, other MMEs 2033, an SGW (Serving Gate Way) 2034, a PGW (PDN Gate Way) 2035, and a PCRF (Policy and Charging Rules Function) 2036. Optionally, MME 2031 is a control node that processes signaling between UE 201 and EPC 203, providing bearer and connection management. HSS 2032 provides registers for managing functions such as the Home Location Register (not shown) and stores user-specific information such as service features and data rates. All user data can be sent through SGW2034, PGW2035 can provide IP address allocation and other functions for UE 201, PCRF2036 is the policy and charging control policy decision point for service data flow and IP bearer resources, and it selects and provides available policy and charging control decisions for the policy and charging execution function unit (not shown in the figure).

[0102] The IP service 204 may include the Internet, an intranet, an IMS (IP Multimedia Subsystem), or other IP services.

[0103] Although the above introduction takes the LTE system as an example, those skilled in the art should know that this application is not only applicable to the LTE system, but can also be applied to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA, 5G and future new network systems (such as 6G), etc., which are not limited here.

[0104] Figure 3This is a schematic diagram of the hardware structure of a controller 140 provided in this application. The controller 140 includes a memory 1401 and a processor 1402. The memory 1401 is used to store program instructions, and the processor 1402 is used to call the program instructions in the memory 1401 to execute the steps performed by the controller in the first embodiment of the above method. The implementation principles and beneficial effects are similar and will not be repeated here.

[0105] Optionally, the controller further includes a communication interface 1403, which can be connected to the processor 1402 via a bus 1404. The processor 1402 can control the communication interface 1403 to implement the receiving and sending functions of the controller 140.

[0106] Figure 4 This is a schematic diagram of the hardware structure of a network node 150 provided in this application. The network node 150 includes a memory 1501 and a processor 1502. The memory 1501 is used to store program instructions, and the processor 1502 is used to call the program instructions in the memory 1501 to execute the steps performed by the first node in the first embodiment of the above method. The implementation principles and beneficial effects are similar and will not be repeated here.

[0107] Optionally, the controller further includes a communication interface 1503, which can be connected to the processor 1502 via a bus 1504. The processor 1502 can control the communication interface 1503 to implement the receiving and sending functions of the network node 150.

[0108] The integrated modules implemented in the form of software function modules can be stored in a computer-readable storage medium. The software function modules stored in a storage medium include a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute some of the steps of the methods of various embodiments of the present application.

[0109] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive solid state disk, SSD), etc.

[0110] Based on the above-mentioned mobile terminal hardware structure and communication network system, various embodiments of the present application are proposed.

[0111] First embodiment

[0112] Reference Figure 5 , Figure 5 FIG. 1 is a flow chart of an image processing method according to a first embodiment. The image processing method according to the embodiment of the present application can be applied to a processing device, including step S10:

[0113] S10: Performing filtering processing on at least one image block according to the spatial domain features of the at least one image block.

[0114] In this embodiment, the processing device can be a smart terminal, such as a mobile phone, a computer, etc., or a server, such as a local server or a cloud server. In this embodiment and this application, the processing device is mainly described as a smart terminal.

[0115] Optionally, the technical solution of this embodiment can be applied to the fields of image coding and decoding, video coding and decoding, hardware video coding and decoding, dedicated circuit video coding and decoding, real-time video coding and decoding, and so on.

[0116] Optionally, for ease of understanding, a brief introduction to the encoding and decoding process is given: Figure 6As shown, it includes modules such as general coding control, transformation and quantization, intra-frame estimation, intra-frame prediction, motion compensation, motion estimation, inverse quantization and inverse transformation, filter control analysis, deblocking filter and SAO filter (i.e., loop filter), entropy coding, decoding frame buffer, and post-filtering (NNPF, Neural Network Post Filter, neural network post-filtering). Optionally, the motion compensation module can perform intra-frame / inter-frame selection to determine specific compensation. Optionally, when performing entropy coding, it is based on the general control data determined by the general coding control module, the variable quantization coefficient determined by the transformation and quantization module, the intra-frame prediction data determined by the filter control analysis, and the motion data determined by the filter control and decoding frame buffer, thereby obtaining the coding bit rate.

[0117] Optionally, the decoded video signal is outputted via a decoded frame buffer.

[0118] Optionally, the loop filter may include two branches: a deblocking filter and an LC-NNLF. The branch results are then fused and SAO (Sample Adaptive Offset) and ALF (Adaptive Loop Filter) processing is performed.

[0119] Optionally, the technical solution of this embodiment can be applied to the filtering stage in the video encoding and / or decoding stage to perform filtering processing on at least one input image block, for example, in the loop filtering stage and / or post-filtering stage.

[0120] Optionally, the image block may be an image block to be filtered, and the image block to be filtered may be, for example, an image block to be filtered in a video encoding and / or decoding stage.

[0121] Optionally, the image block may be an image block obtained after preprocessing the image block to be filtered. For example, the preprocessing may be transformation, quantization, sharpening, etc., which is not limited here.

[0122] Optionally, the image block may include at least one pixel to be filtered.

[0123] Optionally, the pixels to be filtered may be from an image block, or may be an image block obtained after preprocessing the image block to be filtered.

[0124] Optionally, the pixel to be filtered may be a predicted pixel (such as a predicted pixel), a reconstructed pixel (such as a reconstructed pixel), or a reference pixel.

[0125] Optionally, the predicted pixels, reconstructed pixels and / or reference pixels may be intra-frame pixels and / or inter-frame pixels, or may be intermediate values ​​used or generated when the predicted pixels and / or reconstructed pixels generate reference pixels, and no limitation is imposed here.

[0126] Optionally, the spatial domain feature of the image block may include at least one of a style feature, a gradient distribution, an edge direction histogram, a color component correlation, and a motion activity of the image block.

[0127] Optionally, the gradient distribution is such as the gradient calculation of a VVC (Versatile Video Coding, a new generation video coding standard jointly developed by the International Telecommunication Union Telecommunication Standardization Sector and the International Organization for Standardization / International Electrotechnical Commission Moving Picture Experts Group) ALF (Adaptive Loop Filter).

[0128] Optionally, the edge direction histogram corresponds to AV1 (AOMedia Video 1, an open source, royalty-free, next-generation video coding standard developed by the Alliance for Open Media) directional filtering.

[0129] Optionally, the color component correlation covers VVC CCLM (Cross-Component Linear Model) chrominance prediction.

[0130] Optionally, motion activity is used for dynamic QP (quantization parameter) adjustment.

[0131] Optionally, one image block may correspond to one spatial feature, or two or more image blocks may correspond to one spatial feature, or one image block may correspond to two or more spatial features, or two or more image blocks may correspond to two or more spatial features.

[0132] Optionally, each pixel to be filtered in an image block may correspond to at least one spatial feature.

[0133] Optionally, a spatial feature may be determined or obtained based on at least one pixel to be filtered in at least one image block.

[0134] Optionally, the spatial feature of the image block may include at least one spatial feature determined or obtained based on at least one pixel to be filtered in the image block.

[0135] Optionally, a spatial feature can be determined or obtained based on a pixel to be filtered, or two or more spatial features can be determined or obtained based on a pixel to be filtered, or a spatial feature can be determined or obtained based on two or more pixels to be filtered, or two or more spatial features can be determined or obtained based on two or more pixels to be filtered.

[0136] Optionally, the spatial features of an image block may include spatial features corresponding to all or part of the pixels to be filtered in the image block. The spatial features corresponding to a pixel to be filtered may be determined or obtained based on at least one pixel to be filtered in the image block, such as being determined or obtained based on the pixel to be filtered and other pixels to be filtered in the image block.

[0137] Optionally, the spatial domain feature of at least one image block may be determined or obtained based on a statistical feature value of at least one pixel to be filtered in the at least one image block.

[0138] Optionally, the spatial domain features of at least one image block may be determined or obtained by preprocessing the image block and then determining or obtaining the statistical feature value of at least one pixel / element to be filtered in the preprocessed image block.

[0139] Optionally, the statistical characteristic value may include at least one of the mean, maximum, minimum, median, mode (the pixel value with the highest frequency), range (the difference between the maximum and minimum values), variance, standard deviation, etc., or may include values ​​determined or obtained according to other rules or calculation methods, which are not limited here.

[0140] Optionally, the spatial domain features of the image block may be determined or obtained by sequentially calculating statistical feature values ​​of all pixels to be filtered in the image block, and based on these statistical feature values.

[0141] Optionally, filtering processing may be performed on an image block based on the spatial characteristics of the image block, or based on the spatial characteristics of two or more image blocks, or based on the spatial characteristics of the image block. For example, filtering processing may be performed on an image block based on its spatial characteristics.

[0142] Optionally, the image block may include image block features of at least one channel. Optionally, the image block features of at least one channel may be filtered based on the spatial domain features of the image block.

[0143] Optionally, the processing device may be a decoding end. If at the decoding end, the image block may be a decoded image block.

[0144] Optionally, the processing device may be an encoding end. If at the encoding end, the image block may be an image block that has been filtered.

[0145] In this embodiment, by performing filtering processing on at least one image block based on the spatial characteristics of the at least one image block, it is possible to comprehensively consider the spatial characteristics of the at least one image block when performing filtering processing on the at least one image block to determine the key image block features that need to be processed in the image block, thereby focusing on the key image block features during filtering processing, and improving the filtering effect of the filtering processing on the at least one image block.

[0146] Second embodiment

[0147] Based on the first embodiment, a second embodiment is proposed.

[0148] In this embodiment, the spatial feature is determined or obtained according to at least one of the following methods 1 to 8:

[0149] Method 1: statistical characteristic values ​​of a pixel to be filtered and at least one neighboring pixel in at least one image block;

[0150] Optionally, step S1 includes: performing filtering processing on at least one image block according to a spatial feature of the at least one image block, where the spatial feature is determined or obtained based on statistical feature values ​​of a pixel to be filtered and at least one neighboring pixel in the at least one image block.

[0151] Optionally, the statistical characteristic value may include at least one of the mean, maximum value, minimum value, median, mode (the pixel value with the highest frequency), range (the difference between the maximum and minimum values), variance, standard deviation, etc. of the pixel to be filtered and at least one neighboring pixel in at least one image block, or may include a value determined or obtained according to other rules or calculation methods, which is not limited here.

[0152] Optionally, the neighboring pixels may be pixels within a certain range around the pixel to be filtered in the image block. For example, the neighboring pixels may include pixels in at least one of eight directions: directly to the left, directly above, directly to the right, directly below, upper left, upper right, lower left, and lower right of the pixel to be filtered, and are within a preset distance from the pixel to be filtered.

[0153] Optionally, the preset distance may be determined according to the number of pixels contained in the image block. For example, the greater the number of pixels in the image block, the greater the preset distance may be. Optionally, the preset distance may be set to 1 or 2.

[0154] Optionally, the neighboring pixels may include pixels directly adjacent to the pixel to be filtered in the image block, or may include pixels indirectly adjacent to the pixel to be filtered in the image block.

[0155] Optionally, for example, the neighboring pixels may include 8 pixels in the image block that are directly to the left, directly above, directly to the right, directly below, upper left, upper right, lower left and lower right of the pixel to be filtered, or may also include 16 pixels surrounding the 8 pixels.

[0156] Optionally, the spatial characteristics of at least one image block may be determined or obtained based on the mean of the pixel to be filtered and at least one neighboring pixel in at least one image block, and filtering may be performed on at least one image block based on the spatial characteristics of the at least one image block.

[0157] Optionally, the spatial characteristics of at least one image block can be determined or obtained based on the average of the pixel to be filtered in at least one image block and eight pixels to the left, above, to the right, below, above left, above right, below left and below right of the pixel to be filtered, and filtering processing can be performed on the at least one image block based on the spatial characteristics of the at least one image block.

[0158] Optionally, the spatial characteristics of at least one image block can be determined or obtained based on the statistical characteristic values ​​of the pixel to be filtered in at least one image block and at least one neighboring pixel of the pixel to be filtered, and the at least one image block can be filtered based on the spatial characteristics of the at least one image block.

[0159] Optionally, for all or part of the pixels to be filtered in at least one image block, at least one neighboring pixel of the pixel to be filtered in at least one image block can be determined, and statistical characteristic values ​​of the pixel to be filtered and the at least one neighboring pixel can be determined. At least one spatial domain feature of the at least one image block can be determined or obtained based on the statistical characteristic values.

[0160] In this embodiment, by filtering at least one image block based on the spatial characteristics of the at least one image block, it is possible to comprehensively consider the spatial characteristics determined or obtained based on the statistical characteristic values ​​of the pixel to be filtered and at least one neighboring pixel in the at least one image block when filtering the at least one image block, so as to determine the key image block features that need to be processed in the image block, thereby focusing on the key image block features during the filtering process, and improving the filtering effect of the filtering process on the at least one image block.

[0161] Method 2: statistical characteristic values ​​of at least two non-neighboring pixels in at least one image block;

[0162] Optionally, step S1 includes: performing filtering processing on at least one image block according to a spatial feature of the at least one image block, where the spatial feature is determined or obtained based on statistical feature values ​​of at least two non-neighbor pixels in the at least one image block.

[0163] Optionally, the statistical characteristic value may include at least one of the mean, maximum value, minimum value, median, mode (the pixel value with the highest frequency), range (the difference between the maximum and minimum values), variance, standard deviation, etc. of at least two non-neighbor pixels in at least one image block, or may include a value determined or obtained according to other rules or calculation methods, which is not limited here.

[0164] Optionally, the non-neighbor pixels may be pixels other than the pixel to be filtered and the neighboring pixels in the image block. For example, the non-neighbor pixels may be pixels in the image block that are outside a certain range around the pixel to be filtered. For example, the non-neighbor pixels may include pixels that are located in at least one of eight directions: directly to the left, directly above, directly to the right, directly below, upper left, upper right, lower left, and lower right of the pixel to be filtered, and are spaced beyond a preset distance from the pixel to be filtered.

[0165] Optionally, the preset distance may be determined according to the number of pixels contained in the image block. For example, the greater the number of pixels in the image block, the greater the preset distance may be. Optionally, the preset distance may be set to 1 or 2.

[0166] For example, the size of the image block is 7×7, and for the pixel to be filtered that is located in the middle, the non-neighboring pixels may include at least one of the outermost 24 pixels in the image block.

[0167] Optionally, the spatial characteristics of the at least one image block may be determined or obtained based on the statistical characteristic values ​​of at least two non-neighbor pixels in the at least one image block, and filtering processing may be performed on the at least one image block based on the spatial characteristics of the at least one image block.

[0168] Optionally, for all or part of the pixels to be filtered in at least one image block, at least two non-neighboring pixels of the pixel to be filtered in at least one image block can be determined, statistical characteristic values ​​of the at least two non-neighboring pixels can be determined, and at least one spatial domain feature of the at least one image block can be determined or obtained based on the statistical characteristic values.

[0169] In this embodiment, by filtering at least one image block based on the spatial characteristics of the at least one image block, the spatial characteristics determined or obtained based on the statistical characteristic values ​​of at least two non-neighboring pixels in the at least one image block are comprehensively considered to determine the key image block features that need to be processed in the image block, and then the key image block features can be focused during the filtering process, thereby improving the filtering effect of the filtering process on the at least one image block.

[0170] Method three: a statistical feature value of a pixel to be filtered in at least one image block and at least one of the following: an upper neighboring pixel of the pixel to be filtered, an upper neighboring pixel of the pixel to be filtered, an upper neighboring pixel of the pixel to be filtered, an upper neighboring pixel of the pixel to be filtered, and an upper neighboring pixel of the pixel to be filtered;

[0171] Optionally, step S1 includes: filtering at least one image block based on the spatial domain features of at least one image block, where the spatial domain features are determined or obtained based on the statistical characteristic values ​​of the pixel to be filtered in the at least one image block and at least one upper adjacent pixel of the pixel to be filtered, at least one left adjacent pixel of the pixel to be filtered, at least one right adjacent pixel of the pixel to be filtered, and at least one lower adjacent pixel of the pixel to be filtered.

[0172] Optionally, the statistical feature value may include at least one of the mean, maximum value, minimum value, median, mode (the pixel value with the highest frequency), range (the difference between the maximum and minimum values), variance, standard deviation, etc. of the pixel to be filtered in at least one image block and at least one of the upper adjacent pixels of the pixel to be filtered, at least one left adjacent pixel of the pixel to be filtered, at least one right adjacent pixel of the pixel to be filtered, and at least one lower adjacent pixel of the pixel to be filtered, or may include values ​​determined or obtained according to other rules or calculation methods, which are not limited here.

[0173] Optionally, the adjacent pixels may be pixels adjacent to the pixel to be filtered in the image block, and the adjacent pixels of the pixel to be filtered may include at least one of at least one upper adjacent pixel, at least one lower adjacent pixel, at least one right adjacent pixel, and at least one lower adjacent pixel.

[0174] Optionally, the spatial characteristics of at least one image block can be determined or obtained based on the statistical characteristic values ​​of the pixel to be filtered in at least one image block and at least one upper adjacent pixel of the pixel to be filtered, at least one left adjacent pixel of the pixel to be filtered, at least one right adjacent pixel of the pixel to be filtered, and at least one lower adjacent pixel of the pixel to be filtered, and filtering processing can be performed on the at least one image block based on the spatial characteristics of the at least one image block.

[0175] Optionally, for all or part of the pixels to be filtered in at least one image block, a statistical characteristic value of the pixel to be filtered in at least one image block and at least one of at least one upper adjacent pixel of the pixel to be filtered, at least one left adjacent pixel of the pixel to be filtered, at least one right adjacent pixel of the pixel to be filtered, and at least one lower adjacent pixel of the pixel to be filtered can be determined, and at least one spatial feature of the at least one image block can be determined or obtained based on the statistical characteristic value.

[0176] Optionally, the spatial characteristics of at least one image block can be determined or obtained based on the statistical characteristic values ​​of the pixel to be filtered in at least one image block and an upper adjacent pixel, a left adjacent pixel, a right adjacent pixel and a lower adjacent pixel of the pixel to be filtered, and the at least one image block is filtered based on the spatial characteristics of the at least one image block.

[0177] In this embodiment, filtering processing is performed on at least one image block based on the spatial features of the at least one image block. Comprehensive consideration is given to spatial features determined or obtained based on statistical feature values ​​of a pixel to be filtered and at least one of at least one upper neighboring pixel of the pixel to be filtered, at least one left neighboring pixel of the pixel to be filtered, at least one right neighboring pixel of the pixel to be filtered, and at least one lower neighboring pixel of the pixel to be filtered. Key image block features that need to be processed in the image block are determined, and thus the key image block features can be focused on during filtering processing, thereby improving the filtering effect of the filtering processing on the at least one image block.

[0178] Method 4: weighted statistical feature values ​​of the pixel to be filtered and its neighboring pixels in at least one image block;

[0179] Optionally, step S1 includes: filtering at least one image block according to a spatial domain feature of at least one image block, where the spatial domain feature is determined or obtained based on weighted statistical feature values ​​of a pixel to be filtered and pixels adjacent to the pixel to be filtered in the at least one image block.

[0180] Optionally, the concept of adjacent pixels may refer to the above-mentioned method three.

[0181] Optionally, the weighted statistical feature value may include at least one of the weighted mean, weighted maximum, weighted minimum, weighted median, weighted mode (the pixel value with the highest frequency), weighted range (the difference between the maximum and minimum values), weighted variance, weighted standard deviation, etc. of the pixels to be filtered and adjacent to the pixels to be filtered in at least one image block, or may include values ​​determined or obtained according to other rules or calculation methods, which are not limited here.

[0182] Optionally, the spatial characteristics of at least one image block can be determined or obtained based on the weighted statistical characteristic values ​​of the pixel to be filtered in at least one image block and at least one upper adjacent pixel of the pixel to be filtered, at least one left adjacent pixel of the pixel to be filtered, at least one right adjacent pixel of the pixel to be filtered, and at least one lower adjacent pixel of the pixel to be filtered, and filtering processing is performed on the at least one image block based on the spatial characteristics of the at least one image block.

[0183] Optionally, the weights of the weighting can be preset.

[0184] Optionally, the weighted weights can be obtained through pre- / offline neural network training.

[0185] Optionally, the weights of different pixels may be the same or different.

[0186] Optionally, for all or part of the pixels to be filtered in at least one image block, a weighted statistical eigenvalue of the pixel to be filtered in at least one image block and at least one of the upper adjacent pixels of the pixel to be filtered, at least one left adjacent pixel of the pixel to be filtered, at least one right adjacent pixel of the pixel to be filtered, and at least one lower adjacent pixel of the pixel to be filtered can be determined, and at least one spatial feature of the at least one image block can be determined or obtained based on the weighted statistical eigenvalue.

[0187] Optionally, the spatial characteristics of at least one image block can be determined or obtained based on the weighted statistical characteristic values ​​of the pixel to be filtered in at least one image block and an upper adjacent pixel, a left adjacent pixel, a right adjacent pixel and a lower adjacent pixel of the pixel to be filtered, and the at least one image block is filtered based on the spatial characteristics of the at least one image block.

[0188] In this embodiment, by filtering at least one image block based on the spatial characteristics of the at least one image block, the spatial characteristics determined or obtained based on the weighted statistical characteristic values ​​of the pixel to be filtered and the pixels adjacent to the pixel to be filtered in the at least one image block are comprehensively considered to determine the key image block features that need to be processed in the image block, and thus the key image block features can be focused during the filtering process, thereby improving the filtering effect of the filtering process on the at least one image block.

[0189] Method 5: Statistical characteristic value of at least one pixel in the outermost layer of a window centered on the pixel to be filtered, where the size parameter of the window is less than or equal to the size parameter of the image block;

[0190] Optionally, step S1 includes: filtering at least one image block based on the spatial domain features of at least one image block, the spatial domain features are determined or obtained based on the statistical characteristic value of at least one pixel in the outermost layer of a window centered on the pixel to be filtered, and the size parameter of the window is less than or equal to the size parameter of the image block.

[0191] Optionally, the statistical characteristic value may include at least one of the mean, maximum value, minimum value, median, mode (the pixel value with the highest frequency), range (the difference between the maximum and minimum values), variance, standard deviation, etc. of at least one pixel in the outermost layer of the window centered on the pixel to be filtered, or may include a value determined or obtained according to other rules or calculation methods, which is not limited here.

[0192] Optionally, the window may be a window centered on the pixel to be filtered, and a size parameter of the window may be set to be smaller than or equal to a size parameter of the image block.

[0193] Optionally, the spatial characteristics of at least one image block can be determined or obtained based on the statistical characteristic value of at least one pixel in the outermost layer of a window centered on the pixel to be filtered in at least one image block, and filtering processing can be performed on at least one image block based on the spatial characteristics of the at least one image block.

[0194] Optionally, for all or part of the pixels to be filtered in at least one image block, the statistical characteristic value of at least one pixel in the outermost layer of a window centered on the pixel to be filtered in at least one image block can be determined, and at least one spatial feature of the at least one image block can be determined or obtained based on the statistical characteristic value.

[0195] For example, the size parameter of the image block is 7×7, and the size parameter of the window can be set to 5×5. For the pixel to be filtered in the middle of the image block, the spatial domain feature of at least one image block can be determined or obtained based on the statistical feature value of at least one pixel in the outermost 16 pixels of the 5×5 window centered on the pixel to be filtered.

[0196] Optionally, the spatial domain features of at least one image block can be determined or obtained based on the weighted statistical feature value of at least one pixel in the outermost layer of a window centered on the pixel to be filtered in at least one image block, and filtering processing can be performed on at least one image block based on the spatial domain features of the at least one image block.

[0197] In this embodiment, filtering is performed on at least one image block based on the spatial characteristics of the at least one image block, and spatial characteristics determined or obtained based on the statistical characteristic values ​​of at least one pixel in the outermost layer of a window centered on the pixel to be filtered are comprehensively considered to determine key image block features that need to be processed in the image block. This allows the key image block features to be focused during filtering, thereby improving the filtering effect of the filtering process on the at least one image block.

[0198] Method six: statistical eigenvalues ​​of eight pixels that are not adjacent to the pixel to be filtered;

[0199] Optionally, step S1 includes: performing filtering processing on at least one image block according to a spatial domain feature of the at least one image block, where the spatial domain feature is determined or obtained based on statistical characteristic values ​​of eight pixels that are not adjacent to the pixel to be filtered.

[0200] Optionally, the statistical characteristic value may include at least one of the mean, maximum value, minimum value, median, mode (the pixel value with the highest frequency), range (the difference between the maximum and minimum values), variance, standard deviation, etc. of eight pixels that are not adjacent to the pixel to be filtered, or may include values ​​determined or obtained according to other rules or calculation methods, which are not limited here.

[0201] Optionally, eight pixels can be determined from pixels in at least one image block that are not adjacent to the pixel to be filtered, and the spatial characteristics of the at least one image block can be determined or obtained based on the statistical characteristic values ​​of the determined eight pixels, and the at least one image block can be filtered based on the spatial characteristics of the at least one image block.

[0202] Optionally, for all or part of the pixels to be filtered in at least one image block, statistical characteristic values ​​in at least one image block that are not adjacent to the pixels to be filtered can be determined, and at least one spatial feature of the at least one image block can be determined or obtained based on the statistical characteristic values.

[0203] Optionally, for example, the size of the image block is 5×5. For the middlemost pixel to be filtered in the image block, 8 pixels can be determined from 16 pixels that are not adjacent to the pixel to be filtered. Based on the statistical characteristic values ​​of the determined 8 pixels, at least one spatial domain feature of at least one image block can be determined or obtained.

[0204] Optionally, eight pixels can be determined from pixels in at least one image block that are not adjacent to the pixel to be filtered, and the spatial characteristics of the at least one image block can be determined or obtained based on the weighted statistical characteristic values ​​of the determined eight pixels. The at least one image block is filtered based on the spatial characteristics of the at least one image block, and the weighted weights of each pixel can be the same or different.

[0205] Optionally, the eight pixels that are not adjacent to the pixel to be filtered may not be adjacent to each other. For example, if the size of the image block is 5×5, for the center pixel to be filtered in the image block, eight non-adjacent pixels may be determined from the 16 pixels that are not adjacent to the pixel to be filtered. At least one spatial feature of at least one image block may be determined or obtained based on the statistical feature values ​​of the determined eight non-adjacent pixels.

[0206] In this embodiment, at least one image block is filtered based on the spatial characteristics of the at least one image block, and the spatial characteristics determined or obtained based on the statistical characteristic values ​​of eight pixels that are not adjacent to the pixel to be filtered are comprehensively considered to determine the key image block features that need to be processed in the image block. Therefore, the key image block features can be focused during the filtering process, thereby improving the filtering effect of the filtering process on the at least one image block.

[0207] Method seven, statistical characteristic values ​​of a sub-block to be filtered that contains at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel in at least one image block;

[0208] Optionally, step S1 includes: filtering at least one image block based on the spatial domain features of at least one image block, where the spatial domain features are determined or obtained based on the statistical characteristic values ​​of the sub-block to be filtered that contains at least one pixel to be filtered, at least one neighboring pixel and at least one non-neighboring pixel in the at least one image block.

[0209] Optionally, the statistical characteristic value may include at least one of the mean, maximum value, minimum value, median, mode (the pixel value with the highest frequency), range (the difference between the maximum and minimum values), variance, standard deviation, etc. of the sub-block to be filtered that contains at least one pixel to be filtered, at least one neighboring pixel and at least one non-neighboring pixel in at least one image block, or may include values ​​determined or obtained according to other rules or calculation methods, which are not limited here.

[0210] Optionally, the concept of neighboring pixels may refer to the above-mentioned method 1, and the concept of non-neighboring pixels may refer to the above-mentioned method 2.

[0211] Optionally, the image block may include at least one sub-block to be filtered, and the sub-block to be filtered may include at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel.

[0212] Optionally, each sub-block to be filtered in an image block may correspond to at least one spatial feature.

[0213] Optionally, a spatial feature of an image block may be determined or obtained based on at least one sub-block to be filtered in at least one image block. Optionally, a spatial feature of an image block may include at least one spatial feature determined or obtained based on at least one sub-block to be filtered in the image block.

[0214] Optionally, the sub-block to be filtered may include at least one pixel to be filtered and at least one neighboring pixel of the pixel to be filtered, or the sub-block to be filtered may include at least one pixel to be filtered and at least one non-neighboring pixel of the pixel to be filtered, or the sub-block to be filtered may include at least one pixel to be filtered, at least one neighboring pixel of the pixel to be filtered, and at least one non-neighboring pixel of the pixel to be filtered.

[0215] Optionally, a sub-block to be filtered that includes at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel can be determined from the image block, and the spatial domain features of at least one image block can be determined or obtained based on the statistical characteristic values ​​of the sub-block to be filtered, and filtering processing can be performed on at least one image block based on the spatial domain features of the at least one image block.

[0216] Optionally, for all or part of the pixels to be filtered in at least one image block, a sub-block to be filtered in the at least one image block containing at least one of the pixel to be filtered, at least one neighboring pixel of the pixel to be filtered, and at least one non-neighboring pixel of the pixel to be filtered can be determined, and the spatial domain characteristics of the at least one image block can be determined or obtained based on the statistical characteristic values ​​of the sub-block to be filtered, and filtering processing can be performed on the at least one image block based on the spatial domain characteristics of the at least one image block.

[0217] Optionally, a sub-block to be filtered that includes at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel can be determined from the image block, and the spatial domain features of at least one image block can be determined or obtained based on the weighted statistical feature values ​​of the sub-block to be filtered, and filtering processing can be performed on the at least one image block based on the spatial domain features of the at least one image block.

[0218] In this embodiment, by filtering at least one image block based on the spatial characteristics of the at least one image block, comprehensive consideration is given to the spatial characteristics determined or obtained based on the statistical characteristic values ​​of the sub-block to be filtered that includes at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel, so as to determine the key image block features that need to be processed in the image block. This allows the key image block features to be focused during the filtering process, thereby improving the filtering effect of the filtering process on the at least one image block.

[0219] In the eighth approach, at least one image block includes a statistical characteristic value of at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel in the area to be filtered.

[0220] Optionally, step S1 includes: filtering at least one image block based on the spatial domain features of at least one image block, where the spatial domain features are determined or obtained based on the statistical characteristic values ​​of the to-be-filtered area in the at least one image block containing at least one to-be-filtered pixel, at least one neighboring pixel, and at least one non-neighboring pixel.

[0221] Optionally, the statistical characteristic value may include at least one of the mean, maximum value, minimum value, median, mode (the pixel value with the highest frequency), range (the difference between the maximum and minimum values), variance, standard deviation, etc. of the area to be filtered that contains at least one pixel to be filtered, at least one neighboring pixel and at least one non-neighboring pixel in at least one image block, or may include values ​​determined or obtained according to other rules or calculation methods, which are not limited here.

[0222] Optionally, the concept of neighboring pixels may refer to the above-mentioned method 1, and the concept of non-neighboring pixels may refer to the above-mentioned method 2.

[0223] Optionally, the image block may include at least one area to be filtered, and the area to be filtered may include at least one sub-block to be filtered. The concept of the sub-block to be filtered may refer to the above-mentioned method seven.

[0224] Optionally, each area to be filtered in an image block may correspond to at least one spatial feature.

[0225] Optionally, a spatial feature of an image block may be determined or obtained based on at least one region to be filtered in at least one image block.

[0226] Optionally, the spatial domain feature of the image block may include at least one spatial domain feature determined or obtained based on at least one area to be filtered in the image block.

[0227] Optionally, an area to be filtered that includes at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel can be determined from the image block, and the spatial characteristics of at least one image block can be determined or obtained based on the statistical characteristic values ​​of the area to be filtered, and filtering processing can be performed on at least one image block based on the spatial characteristics of the at least one image block.

[0228] Optionally, for all or part of the pixels to be filtered in at least one image block, a region to be filtered in the at least one image block can be determined, which includes at least one of the pixel to be filtered, at least one neighboring pixel of the pixel to be filtered, and at least one non-neighboring pixel of the pixel to be filtered. The spatial domain features of the at least one image block can be determined or obtained based on the statistical characteristic values ​​of the region to be filtered, and filtering processing can be performed on the at least one image block based on the spatial domain features of the at least one image block.

[0229] Optionally, an area to be filtered that includes at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel can be determined from the image block, and the spatial domain features of at least one image block can be determined or obtained based on the weighted statistical feature values ​​of the area to be filtered, and filtering processing can be performed on at least one image block based on the spatial domain features of the at least one image block.

[0230] In this embodiment, by filtering at least one image block based on the spatial characteristics of the at least one image block, the spatial characteristics determined or obtained based on the statistical characteristic values ​​of the to-be-filtered area including at least one of at least one to-be-filtered pixel, at least one neighboring pixel, and at least one non-neighboring pixel are comprehensively considered to determine the key image block features that need to be processed in the image block, thereby focusing on the key image block features during the filtering process, and improving the filtering effect of the filtering process on the at least one image block.

[0231] Third embodiment

[0232] Based on any of the above embodiments, a third embodiment is proposed.

[0233] In this embodiment, the image processing method, step S1, performs filtering processing on at least one image block based on the spatial domain features of at least one image block, including at least one of the ninth to eleventh methods:

[0234] Method nine: determining or obtaining at least one spatial attention weight based on at least one spatial feature and at least one lookup table, and performing filtering processing on an intermediate element block corresponding to at least one image block based on the at least one spatial attention weight;

[0235] Optionally, at least one spatial attention weight can be determined or obtained based on at least one spatial feature of at least one image block and at least one lookup table, and the intermediate element block corresponding to at least one image block can be filtered based on the at least one spatial attention weight.

[0236] Optionally, the spatial attention weight can be a self-attention weight at the spatial level.

[0237] Optionally, the intermediate element block may include at least one intermediate element, which may be an element determined based on the pixel to be filtered in the image block, or a filtered pixel, which may be a pixel obtained by filtering the pixel to be filtered.

[0238] Optionally, a pre-filtering process may be performed on at least one pixel to be filtered in at least one image block to obtain an intermediate element block including at least one intermediate element.

[0239] Optionally, the intermediate elements may be obtained by preprocessing the pixels to be filtered.

[0240] Alternatively, lookup tables are a common method for accelerating computation in embedded systems. For a complex function or series of calculations, if the output values ​​are stored in a lookup table, all that needs to be done later is retrieve the values ​​without having to perform the calculations. Therefore, lookup tables are effective when computation time is longer than memory access time.

[0241] Optionally, the lookup table can be used to approximate the calculation of a neural network or a neural network module, such as a 3x3 convolutional layer. The neural network or the neural network module is pre-trained, and the lookup table is obtained based on the input-output mapping of the trained model.

[0242] Optionally, the lookup table may include at least one spatial feature, and at least one spatial attention weight, and a correspondence between the two.

[0243] Optionally, the lookup table may also include at least one spatial feature, at least one intermediate value, and a corresponding relationship between the two.

[0244] Optionally, the lookup table may also include at least one intermediate value, at least one spatial attention weight, and a corresponding relationship between the two.

[0245] Optionally, at least one spatial attention weight may be determined or obtained in at least one lookup table based on at least one spatial feature of at least one image block.

[0246] Optionally, after determining at least one spatial feature of at least one image block, the at least one spatial feature can be input into at least one lookup table for search. Since the lookup table includes the correspondence between at least one spatial feature and at least one spatial attention weight, the spatial attention weight that has a corresponding relationship with the at least one input spatial feature can be searched in the at least one lookup table and output, and then the intermediate element block corresponding to the at least one image block is filtered according to the output at least one spatial attention weight.

[0247] Optionally, at least one spatial feature may be input into at least one lookup table for search to obtain at least one intermediate value, and at least one spatial attention weight may be determined or obtained based on the at least one intermediate value. For example, the at least one intermediate value may be input into at least one neural network and / or at least one activation function to output at least one spatial attention weight.

[0248] Optionally, at least one intermediate value can be determined or obtained based on at least one spatial feature, for example, at least one spatial feature is input into at least one neural network and / or at least one activation function, at least one intermediate value is output, and the intermediate value is input into at least one lookup table for search to obtain at least one spatial attention weight.

[0249] Optionally, the intermediate element block corresponding to the image block may include image block features of at least one channel.

[0250] Optionally, based on the spatial features of an image block and at least one lookup table, the spatial attention weight corresponding to the image block features of at least one channel can be determined or obtained, and the image block features of at least one channel can be filtered based on the spatial attention weight corresponding to the image block features of one channel.

[0251] In this embodiment, at least one spatial attention weight is determined or obtained based on at least one spatial feature of at least one image block and at least one lookup table, and filtering processing is performed on the intermediate element block corresponding to at least one image block based on the at least one spatial attention weight. The key image block features that need to be processed in the image block can be determined through spatial attention, and the key image block features can be focused during filtering processing, which can improve the filtering effect of filtering processing on at least one image block.

[0252] Method 10: determining or obtaining at least one spatial attention weight based on at least one spatial feature and at least one neural network, and performing filtering processing on an intermediate element block corresponding to at least one image block based on the at least one spatial attention weight;

[0253] Optionally, at least one spatial attention weight can be determined or obtained based on at least one spatial feature of at least one image block and at least one neural network, and the intermediate element block corresponding to at least one image block can be filtered based on the at least one spatial attention weight.

[0254] Optionally, the concept of the intermediate element block may refer to the above-mentioned method nine.

[0255] Optionally, the spatial attention weight can be a self-attention weight at the spatial level.

[0256] Optionally, the neural network in this embodiment can be a neural network based on a fully connected layer, a neural network based on a convolutional layer, a neural network based on a Transformer, a neural network based on a hybrid convolutional layer, a fully connected layer and a Transformer, etc.

[0257] Optionally, at least one spatial domain feature may be input into at least one neural network, and at least one spatial attention weight may be output.

[0258] Optionally, at least one spatial feature may be input into at least one neural network, at least one intermediate value may be output, and at least one spatial attention weight may be determined or obtained based on the at least one intermediate value. For example, at least one intermediate value may be input into at least one lookup table or at least one activation function, and at least one spatial attention weight may be output.

[0259] Optionally, at least one intermediate value can be determined or obtained based on at least one spatial feature, for example, at least one spatial feature is input into at least one lookup table or at least one activation function, at least one intermediate value is output, the intermediate value is input into at least one neural network, and at least one spatial attention weight is output.

[0260] Optionally, at least one spatial domain feature may be input into at least one lookup table for search, an intermediate value may be output, and the intermediate value may be input into at least one neural network to output at least one spatial attention weight.

[0261] Optionally, the lookup table may include at least one spatial feature, at least one intermediate value, and a correspondence between the two. Since the lookup table includes the correspondence between the at least one spatial feature and the at least one intermediate value, the intermediate value corresponding to the at least one input spatial feature may be searched in the at least one lookup table and output. The at least one intermediate value output is then input into at least one neural network to obtain at least one spatial attention weight. Filtering processing is then performed on the intermediate element block corresponding to the at least one image block based on the at least one spatial attention weight.

[0262] Optionally, at least one spatial domain feature may be input into at least one neural network, an intermediate value may be output, and the intermediate value may be input into at least one lookup table for search, and at least one spatial attention weight may be output.

[0263] Optionally, the lookup table may include at least one intermediate value, at least one spatial attention weight, and a corresponding relationship between the two. Since the lookup table includes the corresponding relationship between the at least one intermediate value and the at least one spatial attention weight, after inputting the at least one spatial feature into the at least one neural network and outputting the intermediate value, the at least one intermediate value may be input into the at least one lookup table to determine at least one spatial attention weight that corresponds to the input at least one intermediate value, and filtering processing may be performed on the intermediate element block corresponding to the at least one image block based on the at least one spatial attention weight.

[0264] In this embodiment, at least one spatial attention weight is determined or obtained based on at least one spatial feature of at least one image block and at least one neural network, and filtering processing is performed on the intermediate element block corresponding to the at least one image block based on the at least one spatial attention weight. The key image block features that need to be processed in the image block can be determined through spatial attention, and the key image block features can be focused during filtering processing, which can improve the filtering effect of filtering processing on at least one image block.

[0265] Method eleven: determine or obtain at least one spatial attention weight based on at least one spatial feature and at least one activation function, and filter the intermediate element block corresponding to at least one image block based on the at least one spatial attention weight.

[0266] Optionally, at least one spatial attention weight can be determined or obtained based on at least one spatial feature and at least one activation function of at least one image block, and the intermediate element block corresponding to at least one image block can be filtered based on the at least one spatial attention weight.

[0267] Optionally, the concept of the intermediate element block may refer to the above-mentioned method nine.

[0268] Optionally, the spatial attention weight can be a self-attention weight at the spatial level.

[0269] Optionally, an activation function, also known as an excitation function, often exists between the input layer and the output layer of a neural network. Its function is to add some nonlinear factors to the neural network.

[0270] Optionally, at least one spatial feature may be input into at least one activation function, and at least one spatial attention weight may be output.

[0271] Optionally, at least one spatial feature may be input into at least one activation function, outputting at least one intermediate value, and at least one spatial attention weight may be determined or obtained based on the at least one intermediate value. For example, at least one intermediate value may be input into at least one lookup table or at least one neural network, and at least one spatial attention weight may be output.

[0272] Optionally, at least one intermediate value can be determined or obtained based on at least one spatial feature, for example, at least one spatial feature is input into at least one lookup table or at least one neural network, at least one intermediate value is output, the intermediate value is input into at least one activation function, and at least one spatial attention weight is output.

[0273] Optionally, at least one spatial attention weight may be determined or obtained based on at least one spatial feature, at least one activation function, and at least one lookup table.

[0274] Optionally, at least one spatial domain feature may be input into at least one lookup table for search, an intermediate value may be output, and the intermediate value may be input into at least one activation function to output at least one spatial attention weight.

[0275] Optionally, the lookup table may include at least one spatial feature, at least one intermediate value, and a correspondence between the two. Since the lookup table includes the correspondence between the at least one spatial feature and the at least one intermediate value, the intermediate value that corresponds to the at least one input spatial feature can be searched in the at least one lookup table and output. The at least one intermediate value output is then input into at least one activation function to output at least one spatial attention weight.

[0276] Optionally, at least one spatial domain feature may be input into at least one activation function, an intermediate value may be output, and the intermediate value may be input into at least one lookup table for search, and at least one spatial attention weight may be output.

[0277] Optionally, the lookup table may include at least one intermediate value, at least one spatial attention weight, and a corresponding relationship between the two. Since the lookup table includes the corresponding relationship between the at least one intermediate value and the at least one spatial attention weight, after inputting the at least one spatial feature into the at least one activation function and outputting the intermediate value, the at least one intermediate value may be input into the at least one lookup table to search for and determine at least one spatial attention weight that has a corresponding relationship with the input at least one intermediate value.

[0278] Optionally, at least one spatial attention weight may be determined or obtained based on at least one spatial feature, at least one activation function, and at least one neural network.

[0279] Alternatively, at least one spatial feature may be input into at least one activation function, outputting an intermediate value, which is then input into at least one neural network to obtain at least one spatial attention weight. Alternatively, at least one spatial feature may be input into at least one neural network to obtain an intermediate value, which is then input into at least one activation function to obtain at least one spatial attention weight.

[0280] Optionally, at least one spatial attention weight may be determined or obtained based on at least one spatial feature, at least one activation function, at least one neural network, and at least one lookup table.

[0281] In this embodiment, at least one spatial attention weight is determined or obtained based on at least one spatial feature and at least one activation function of at least one image block, and the intermediate element block corresponding to the at least one image block is filtered based on the at least one spatial attention weight. The key image block features that need to be processed in the image block can be determined through spatial attention, and the key image block features can be focused during filtering, which can improve the filtering effect of filtering at least one image block.

[0282] Fourth embodiment

[0283] Based on any of the above embodiments, a fourth embodiment is proposed.

[0284] In this embodiment, the image processing method further includes the following method 12:

[0285] Method 12: Determine or obtain at least one intermediate element block based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block.

[0286] Optionally, at least one intermediate element block can be determined or obtained based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block, and the intermediate element block corresponding to the at least one image block can be filtered based on the spatial domain features of the at least one image block.

[0287] Optionally, the twelfth method can be combined with any one of the above-mentioned methods 1 to 11. For example, the twelfth method is combined with the above-mentioned method 9, and step S1 includes determining or obtaining at least one intermediate element block based on at least one neural network, at least one item in the lookup table, and at least one image block; determining or obtaining at least one spatial attention weight based on at least one spatial feature of the at least one image block and at least one lookup table; and filtering the intermediate element block corresponding to the at least one image block based on the at least one spatial attention weight.

[0288] Optionally, the concept of the neural network may refer to the method 10 in the above embodiment, and the concept of the lookup table may refer to the method 9 in the above embodiment.

[0289] Optionally, the lookup table structure may include at least one lookup table structure, such as a serial structure including at least one lookup table, a parallel structure including at least one lookup table, a tree structure including at least one lookup table, etc., which is not limited here.

[0290] Optionally, the lookup table can be a feature value lookup table, similar to a neural network, where the feature value reflects the distribution characteristics of the original pixels to be filtered, such as edge / texture / semantic intensity. It can also be an attention weight lookup table or a style feature lookup table. The lookup table can be a filtering mode, a filtering pixel lookup table, or a lookup table that needs to be applied in the intermediate process of filtering processing, etc.

[0291] Optionally, filtering processing may be performed on at least one pixel to be filtered in at least one image block according to at least one item among a neural network, a lookup table structure and a lookup table, and at least one intermediate element block may be determined or obtained according to the filtering result.

[0292] In this embodiment, at least one intermediate element block can be determined or obtained based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block. The intermediate element block corresponding to the at least one image block is filtered based on the spatial domain features of the at least one image block. The key image block features that need to be processed in the image block can be determined through spatial attention, and the key image block features can be focused during the filtering process, thereby improving the filtering effect of the filtering process on the at least one image block.

[0293] Optionally, the lookup table structure includes at least one of the following modes 13 to 29:

[0294] Method 13, at least one lookup table;

[0295] Optionally, the lookup table structure may include at least one lookup table, or may include at least one lookup table and at least one neural network module.

[0296] Optionally, an index corresponding to at least one pixel to be filtered may be determined, and the index may be input into at least one lookup table in the lookup table structure for search, and at least one intermediate element in the intermediate element block may be determined or obtained based on the search result.

[0297] In this embodiment, when the lookup table structure is at least one lookup table, at least one lookup table can be used to perform filtering processing on at least one pixel to be filtered, and then the lookup table can be selected for filtering processing according to different scenarios to improve the filtering effect of the filtering processing, thereby supporting the improvement of the efficiency of video encoding and / or decoding.

[0298] Method 14, at least one neural network module;

[0299] Optionally, the neural network module can be a neural network, such as a neural network that only includes a 3x3 convolutional layer, or a neural network module that includes at least one convolutional layer.

[0300] Optionally, the lookup table structure may include at least one neural network module, or may include at least one lookup table and at least one neural network module at the same time.

[0301] Optionally, at least one pixel to be filtered may be input into a neural network module, and at least one intermediate element in the intermediate element block may be output.

[0302] Optionally, the index corresponding to at least one pixel to be filtered can be determined, and the index can be input into at least one lookup table in the lookup table structure for search, and the search result can be input into the neural network module for processing, and at least one intermediate element in at least one intermediate element block can be determined or obtained based on the output of the neural network module.

[0303] Optionally, at least one pixel to be filtered can be input into a neural network module, at least one index can be determined based on the output result of the neural network module, and the at least one index can be input into at least one lookup table for search, and at least one intermediate element in at least one intermediate element block can be determined or obtained based on the search result.

[0304] In this embodiment, when the lookup table structure is at least one neural network module, at least one neural network module can be used to perform filtering processing on at least one pixel to be filtered, and then the neural network module can be selected for filtering processing according to different scenarios to improve the filtering effect of the filtering processing, thereby supporting the improvement of the efficiency of video encoding and / or decoding.

[0305] Mode 15: at least two search branches, at least one search branch having the same input as another search branch;

[0306] Optionally, the lookup table structure may include at least two lookup branches, and each lookup branch may include at least one lookup table. The at least one lookup table included in each of the at least two lookup branches may be the same or different, which is not limited here.

[0307] Optionally, inputs of at least two search branches are the same, such as at least one of an input pixel value and an input pixel number.

[0308] Optionally, the inputs of at least one lookup table in at least one search branch and at least one lookup table in another search branch may be the same, for example, both come from the same filtering intermediate element block (such as the block to be filtered, or the reference image block determined or obtained based on the block to be filtered).

[0309] For example, the input of at least one lookup table in at least one search branch is two pixels, and the input of at least one lookup table in another search branch can also be two pixels. The two input pixels can be pixels to be filtered, or they can be filtering intermediate elements determined or obtained based on the pixels to be filtered, etc.

[0310] In this embodiment, when the inputs of at least two search branches in the lookup table structure are the same, multiple filtering processes can be performed based on at least one pixel to be filtered in combination with at least two search branches with the same inputs in the lookup table structure, thereby improving the filtering effect.

[0311] Mode 16: at least two search branches, the input of at least one search branch is determined or obtained based on the output of another search branch;

[0312] Optionally, there are at least two search branches in the lookup table structure, and the inputs and outputs of the two search branches may have a certain correlation relationship.

[0313] Optionally, the input of at least one search branch can be determined or obtained based on the output of another search branch. For example, the output of at least one search branch can be used as the input of another search branch, the output range of at least one search branch can be used as the input range of another search branch, etc.

[0314] Optionally, the input of at least one lookup table in at least one search branch can be determined or obtained based on the output of at least one lookup table in another search branch. For example, the output of at least one lookup table in at least one search branch can be used as the input of at least one lookup table in another search branch, and the output range of at least one lookup table in at least one search branch can be used as the input range of at least one lookup table in another search branch, etc.

[0315] Optionally, the input of at least one search branch can be determined or obtained based on at least one pixel to be filtered, for example, at least one pixel to be filtered is used as the input of at least one search branch for filtering, or the index corresponding to at least one pixel to be filtered is used as the input of at least one search branch for filtering, and then the output of the at least one search branch is used as the input of another search branch for filtering, until at least one intermediate element in at least one intermediate element block is finally obtained.

[0316] In this embodiment, when the input of at least one search branch in the at least two search branches of the lookup table structure is determined or obtained based on the output of another search branch, at least one pixel to be filtered can be filtered in sequence according to the at least two search branches, thereby realizing multiple filtering processes and improving the filtering effect.

[0317] Mode 17: at least two search branches, the at least two search branches being located in at least one channel corresponding to at least one pixel to be filtered or the first filtered intermediate element;

[0318] Optionally, the channels in this embodiment may be applicable to a lookup table.

[0319] Optionally, a channel is a component of the feature map of an image block in the depth dimension, which is used to describe the feature representation of the number of features in a specific dimension. Each channel represents a certain feature (such as texture, edge, derivative distribution) extracted from an image block (such as a predicted block or a reconstructed block). For example, one channel may detect horizontal edges, and another channel may detect vertical edges.

[0320] Optionally, at least one pixel to be filtered corresponds to multiple channels.

[0321] Optionally, the first filtering intermediate element may be data generated during the filtering process of the pixel to be filtered, such as a eigenvalue, a filtering mode, or other filtering parameters.

[0322] Optionally, the first filtered intermediate element may belong to a type of filtered intermediate elements.

[0323] Optionally, a filtering intermediate element refers to data generated during the filtering process of a pixel to be filtered. For example, when filtering a pixel to be filtered using three serial lookup tables, the output of the first lookup table and the output of the second lookup table are both filtering intermediate elements. For another example, a new pixel to be filtered obtained after pre-processing the pixel to be filtered by performing a DCT transform can also be called a filtering intermediate element.

[0324] Optionally, at least one first filtered intermediate element corresponds to multiple channels.

[0325] Optionally, in the lookup table structure, there are at least two lookup branches, and the at least two lookup branches are located in the same channel.

[0326] Optionally, the at least two search branches may be used as a filter-like device to perform filtering processing on at least one pixel to be filtered or the first filtering intermediate element of the same channel.

[0327] Optionally, when processing pixel features (such as pixel value, pixel position, etc.) of at least one pixel to be filtered or the first filtered intermediate element in the same channel, at least two search branches can be processed in parallel or in series, and there is no limitation here.

[0328] Optionally, filtering processing may be performed on at least one pixel to be filtered or the first filtering intermediate element in the same channel according to at least one lookup table in each of at least two search branches.

[0329] In this embodiment, by having at least two search branches in the lookup table structure, and at least two search branches being located in at least one channel corresponding to at least one pixel to be filtered or the first filtering intermediate element, the pixel features (such as pixel value or pixel position, etc.) of at least one pixel to be filtered or the first filtering intermediate element in the same channel can be filtered using at least two search branches, thereby embodying the realization of multiple filtering processes on the pixel features of the same channel and improving the filtering effect.

[0330] Mode 18: at least two search branches, the search table search mode of at least two search branches is parallel search and / or serial search;

[0331] Optionally, the lookup table structure includes at least two search branches, and the search mode of the at least two search branches is parallel search and / or serial search.

[0332] Optionally, the input elements corresponding to the lookup tables in at least two search branches can be determined simultaneously based on at least one pixel to be filtered or the first filtering intermediate element, and input into the respective corresponding lookup tables for parallel search to obtain at least one filtering intermediate element, and then a subsequent filtering processing method is used to perform filtering processing to obtain a filtered pixel.

[0333] Optionally, the input element corresponding to the lookup table in at least one search branch can be determined or obtained based on at least one pixel to be filtered or the first filtered intermediate element, and input into the lookup table in the search branch for search. After the search is completed, a search operation of the lookup table in another search branch is performed, that is, a serial search is performed until at least one intermediate element in at least one intermediate element block is finally determined or obtained.

[0334] In this embodiment, by using parallel search and / or serial search as the search method of the lookup table in at least two search branches of the lookup table structure, it is possible to achieve that when filtering at least one pixel to be filtered using at least two search branches of the lookup table structure, different search methods can be selected according to different scenarios to improve the filtering effect.

[0335] Mode 19: at least two lookup tables, wherein the input of at least one lookup table is determined or obtained according to the type, size parameter and / or input range of the other lookup table;

[0336] Optionally, there may be multiple types of lookup tables, the types of the multiple lookup tables may be the same, or the types of the multiple lookup tables may be different, such as a filtered pixel lookup table, a homography matrix lookup table, an eigenvalue lookup table, and the like.

[0337] Optionally, the size parameters of the lookup table may include the width, height, perimeter, and area of ​​the lookup table.

[0338] Optionally, the input range of the lookup table may be set in advance, the input ranges of multiple lookup tables may be the same, or the input ranges of multiple lookup tables may be different.

[0339] Optionally, the lookup table structure includes at least two lookup tables, and the input of one lookup table can be determined according to the type of another lookup table. For example, the types of two consecutive lookup tables are both filtered pixel lookup tables, and the input range of one filtered pixel lookup table is greater than or equal to the input range of the other filtered pixel lookup table.

[0340] Optionally, for at least two lookup tables in the lookup table structure, the input of one lookup table may be determined according to a size parameter of the other lookup table, for example, the input range of the lookup table with a larger size parameter is greater than or equal to the input range of the lookup table with a smaller size parameter.

[0341] Optionally, for at least two lookup tables in the lookup table structure, the input of one lookup table may be determined according to the input range of the other lookup table, for example, the inputs of the two lookup tables may be consistent.

[0342] In this embodiment, by determining or obtaining the input of at least one lookup table according to the type, size parameters and / or input range of another lookup table in at least two lookup tables of the lookup table structure, it is possible to perform multiple filtering processes on at least one pixel to be filtered using at least two lookup tables, thereby improving the filtering effect.

[0343] Mode 20: at least two lookup tables, wherein the at least two lookup tables are located in at least one channel corresponding to at least one pixel to be filtered or the first filtered intermediate element;

[0344] Optionally, at least one pixel to be filtered corresponds to multiple channels, and at least one first filtering intermediate element corresponds to multiple channels.

[0345] Optionally, the lookup table structure may include at least two lookup tables, and the at least two lookup tables may be respectively located in at least one channel of at least one pixel to be filtered, that is, there is at least one lookup table in each channel of at least one pixel to be filtered, or there may be at least two lookup tables in the same channel of at least one pixel to be filtered.

[0346] Optionally, at least two lookup tables can be respectively located in at least one channel of at least one first filtered intermediate element, that is, there is at least one lookup table in each channel of at least one first filtered intermediate element, or there can be at least two lookup tables in the same channel of at least one first filtered intermediate element.

[0347] In this embodiment, by having at least two lookup tables in the lookup table structure, at least two lookup tables are located in at least one channel corresponding to at least one pixel to be filtered or the first filtered intermediate element, it is possible to use at least two lookup tables to perform multiple filtering processes on at least one pixel to be filtered or the first filtered intermediate element of the same channel, thereby improving the filtering effect.

[0348] Mode 21: at least two lookup tables, the search mode of at least two lookup tables in the same channel is parallel search and / or serial search;

[0349] Optionally, in the lookup table structure, the search method of at least two lookup tables is parallel search. When searching at least two lookup tables in parallel, the inputs of the two lookup tables come from the same block (i.e., image block or intermediate element block); by performing a one-time scan on the pixel block to be filtered or the intermediate element block, the indexes of at least two lookup tables can be obtained at the same time, and then the at least two lookup tables can be searched in parallel based on the indexes of the at least two lookup tables.

[0350] Optionally, the intermediate element block may be a block containing filtered intermediate elements. The filtered intermediate elements may refer to the above description and will not be repeated here.

[0351] Optionally, in the lookup table structure, when searching at least two lookup tables in parallel, there is no dependency between the input and output of the two lookup tables, that is, the inputs of the at least two lookup tables can come from different blocks or from the same block; at least two lookup tables can be searched simultaneously by multiple tasks / processes / hardware.

[0352] Optionally, in the lookup table structure, at least two lookup tables are searched in serial mode. When searching at least two lookup tables serially, inputs and outputs of the two lookup tables are dependent on each other, and one lookup table operation must be performed before the second lookup table operation.

[0353] Optionally, the lookup table structure may include at least two lookup tables, and at least two lookup tables exist in the same channel corresponding to at least one pixel to be filtered, and the at least two lookup tables are searched in parallel and / or serially.

[0354] Optionally, at least two lookup tables exist in the same channel corresponding to at least one first filtered intermediate element, and the at least two lookup tables are searched in a parallel manner and / or in a serial manner.

[0355] Optionally, when processing the pixel to be filtered according to the lookup table structure, at least two lookup tables in the same channel can be used to perform parallel and / or serial searches on the index corresponding to at least one pixel to be filtered or the first filtered intermediate element until the filtered pixel is finally obtained.

[0356] In this embodiment, by using parallel search and / or serial search as the search method for at least two lookup tables in the lookup table structure, at least two lookup tables in the same channel can be used. This allows for filtering of at least one pixel to be filtered in the same channel using at least two search branches of the lookup table structure. Different search methods can be selected according to different scenarios to improve the filtering effect.

[0357] Mode 22: at least two lookup tables, and the lookup modes of the lookup tables in at least two channels are parallel search and / or serial search;

[0358] Optionally, the lookup table structure includes at least two lookup tables, and there are lookup tables in at least two channels corresponding to at least one pixel to be filtered, and the lookup tables in these two channels are searched in parallel and / or serially.

[0359] Optionally, there are lookup tables in at least two channels corresponding to at least one first filtered intermediate element, and the lookup tables in the two channels are searched in a parallel search and / or serial search manner.

[0360] Optionally, when processing the pixel to be filtered according to the lookup table structure, the lookup tables in at least two channels can be used to perform parallel searches on the indexes corresponding to the pixel to be filtered, or a serial search method can be used to call the lookup tables in at least two channels in sequence to perform searches until the filtered pixel is finally obtained.

[0361] Optionally, when processing the first filtered intermediate element according to the lookup table structure, the lookup tables in at least two channels can be used to perform parallel searches on the indexes corresponding to the first filtered intermediate element, or a serial search method can be used to call the lookup tables in at least two channels in sequence to perform searches until the filtered pixel is finally obtained.

[0362] In this embodiment, by using parallel search and / or serial search as the search method for the lookup tables in at least two channels in the at least two lookup tables of the lookup table structure, it is possible to achieve that when filtering at least one pixel to be filtered in different channels using at least two search branches of the lookup table structure, different search methods can be selected according to different scenarios to improve the filtering effect.

[0363] Mode 23: at least one search branch and at least one neural network module, wherein the input of the at least one neural network module is the same as the input of the at least one search branch;

[0364] Optionally, the neural network module may include a neural network, such as at least one convolution layer, a 3x3 convolution layer, etc.

[0365] Optionally, an input of at least one search branch in the lookup table structure is the same as an input of at least one neural network module.

[0366] Optionally, at least one pixel to be filtered or the first filtered intermediate element can be simultaneously input into at least one search branch and at least one neural network module for parallel processing, and the output results of the parallel processing of at least one search branch and at least one neural network module can be fused or weighted, and then subsequent filtering processing can be performed until the filtered pixel is finally obtained.

[0367] In this embodiment, when the lookup table structure includes at least one lookup branch and at least one neural network module with the same input, it is possible to use at least one lookup branch of the lookup table and at least one neural network module to jointly perform filtering processing on at least one pixel to be filtered or the first filtered intermediate element, and combine the advantages of both the lookup table in the lookup branch and the neural network in the neural network module to perform filtering processing to improve the filtering effect.

[0368] Mode 24: at least one search branch and at least one neural network module, wherein an input of the at least one neural network module is determined or obtained based on an output of the at least one search branch;

[0369] Optionally, the input of at least one neural network module in the lookup table structure can be determined or obtained based on the output of at least one search branch, and the input of at least one neural network module can be determined or obtained based on the output of at least one lookup table.

[0370] Optionally, the index corresponding to at least one pixel to be filtered or the first filtered intermediate element can be input into the lookup table in at least one search branch for search, and the output of at least one search branch can be obtained, and the output of at least one search branch can be used as the input of at least one neural network module (or the output of at least one search branch can be deformed, such as weighted processing, to obtain the input of at least one neural network module), and input into at least one neural network module for processing until all filtering processing processes are completed and the filtered pixel is obtained.

[0371] In this embodiment, when the lookup table structure includes at least one lookup branch and at least one neural network module, and the input of at least one neural network module is determined or obtained based on the output of at least one lookup branch, it is possible to use at least one lookup branch of the lookup table and at least one neural network module to jointly perform filtering processing on at least one pixel to be filtered or the first filtered intermediate element, and perform filtering processing by combining the advantages of both the lookup table in the lookup branch and the neural network in the neural network module to improve the filtering effect.

[0372] Mode 25: at least one lookup table and at least one neural network module, wherein the input of the at least one neural network module is determined or obtained according to the type, size parameter and / or input range of the at least one lookup table;

[0373] Optionally, the type, size parameters and / or input range of at least one lookup table can refer to the above method ten.

[0374] Optionally, the lookup table structure may include at least one lookup table and at least one neural network module. At least one pixel to be filtered or the first filtered intermediate element may be input into the at least one lookup table for search, and the input of at least one neural network module may be determined or obtained based on the search result, and input into at least one neural network module for processing until all filtering processing processes are completed and the filtered pixel is obtained.

[0375] Optionally, the input of at least one neural network module can be determined based on the type, size parameter and / or input range of at least one lookup table. For example, if the output ranges of two different types of lookup tables are different, their corresponding inputs to at least one neural network module will also be different; for example, if the output ranges of two lookup tables with different size parameters are different, their corresponding inputs to at least one neural network module will also be different.

[0376] In this embodiment, when the lookup table structure includes at least one lookup branch and at least one neural network module, and the input of the at least one neural network module is determined or obtained according to the type, size parameters and / or input range of the at least one lookup table, it is possible to use at least one lookup branch of the lookup table and at least one neural network module to jointly perform filtering processing on at least one pixel to be filtered or the first filtered intermediate element, and combine the advantages of both the lookup table in the lookup branch and the neural network in the neural network module to perform filtering processing to improve the filtering effect.

[0377] Mode 26: at least two lookup tables, wherein the output ranges of the at least two lookup tables are the same and / or the output ranges of the lookup tables are different;

[0378] Optionally, output ranges of at least two lookup tables in the lookup table structure are the same.

[0379] Optionally, at least two lookup tables in the lookup table structure have different output ranges.

[0380] Optionally, there are at least three lookup tables in the lookup table structure, two lookup tables have the same output range, and two lookup tables have different output ranges.

[0381] For example, the lookup table structure includes lookup table 1, lookup table 2, and lookup table 3. Lookup table 1 and lookup table 2 have the same output range, and lookup table 3 has a different output range from lookup table 1 and from lookup table 2.

[0382] Optionally, at least two lookup tables with different output ranges can be used to filter at least one pixel to be filtered or the first filtered intermediate element; at least two lookup tables with the same output range can be used to filter at least one pixel to be filtered or the first filtered intermediate element.

[0383] In this embodiment, by including at least two lookup tables in the lookup table structure, and the output ranges of at least two lookup tables are the same and / or the output ranges of the lookup tables are different, it can be achieved that when filtering at least one pixel to be filtered or the first filtering intermediate element is filtered according to the lookup table structure, filtering can be performed more flexibly according to the output ranges of different lookup tables, thereby improving the filtering effect.

[0384] Mode 27: at least two lookup tables, wherein the input ranges of the at least two lookup tables are the same and / or the input ranges of the lookup tables are different;

[0385] Optionally, at least two lookup tables in the lookup table structure have the same input range.

[0386] Optionally, at least two lookup tables in the lookup table structure have different input ranges.

[0387] Optionally, there are at least three lookup tables in the lookup table structure, two lookup tables have the same input range, and two lookup tables have different input ranges.

[0388] For example, the lookup table structure includes lookup table 1, lookup table 2, and lookup table 3. Lookup table 1 and lookup table 2 have the same input range, and lookup table 3 has a different input range from lookup table 1 and from lookup table 2.

[0389] Optionally, at least two lookup tables with different input ranges can be used to filter at least one pixel to be filtered or the first filtering intermediate element; at least two lookup tables with the same input range can be used to filter at least one pixel to be filtered or the first filtering intermediate element.

[0390] In this embodiment, by including at least two lookup tables in the lookup table structure, and the input ranges of at least two lookup tables are the same and / or the input ranges of the lookup tables are different, it can be achieved that when filtering at least one pixel to be filtered or the first filtering intermediate element is performed according to the lookup table structure, the filtering process can be performed more flexibly according to the input ranges of different lookup tables, thereby improving the filtering effect.

[0391] Mode 28: at least one first residual module based on a lookup table, the first residual module comprising a first branch having at least one lookup table structure, an adder, and a second branch having a short-circuit structure;

[0392] Optionally, a residual structure may be provided in the lookup table structure, that is, at least one first residual module based on the lookup table may be provided, and the residual structure of the lookup table is embodied by the first residual module.

[0393] Optionally, a first residual module can be used to process at least one pixel to be filtered or the first filtered intermediate element, for example, at least one pixel to be filtered or the first filtered intermediate element is processed respectively according to a first branch containing at least one lookup table structure and a second branch containing a short-circuit structure, and then an adder is used to perform weighted addition on the output results of the first branch and the second branch until all filtering processes are completed to obtain a filtered pixel.

[0394] In this embodiment, by including at least one first residual module based on a lookup table in the lookup table structure, the first residual module includes a first branch containing at least one lookup table structure, an adder, and a second branch containing a short-circuit structure, it can be achieved that when filtering at least one pixel to be filtered or the first filtering intermediate element is filtered according to the lookup table structure, the residual structure can be combined for filtering processing to reflect the advantages of the residual structure, thereby improving the filtering effect of the filtering processing.

[0395] Method 29: at least one second residual module based on a lookup table, the first input element of the second residual module passes through a first branch containing at least one lookup table structure to obtain a fourth output element, and the first input element of the third residual module passes through a second branch containing a short-circuit structure to obtain a fifth output element that is the same as the first input element, and the fourth output element and the fifth output element are weightedly added by an adder to obtain a sixth output element.

[0396] Optionally, a residual structure may be provided in the lookup table structure, such as at least one second residual module based on the lookup table, and / or at least one first residual module based on the lookup table.

[0397] Optionally, the first residual module may be the second residual module.

[0398] Optionally, the first input element (such as a pixel or an index mark, etc.) of the second residual module may be determined or obtained according to at least one pixel to be filtered.

[0399] Optionally, in the first branch of the second residual module, the first input element can be input into at least one lookup table structure for processing, such as inputting its corresponding index into the lookup table for search, and the obtained search result is used as the fourth output element of the first branch.

[0400] Optionally, in the second branch of the second residual module, the first input element may be input into the second branch containing the short-circuit result, and a fifth output element identical to the first input element may be output.

[0401] The fourth output element output by the first branch and the fifth output element output by the second branch are input into the adder for weighted addition to obtain a sixth output element.

[0402] Optionally, the sixth output element may be directly output as a filtered pixel, or the sixth output element may be used as a filtering intermediate element to continue subsequent filtering processing until all filtering processes are completed to obtain a filtered pixel.

[0403] In this embodiment, by including at least one second residual module based on the lookup table in the lookup table structure, the first input element of the second residual module passes through the first branch containing at least one lookup table structure to obtain a fourth output element, and the first input element of the third residual module passes through the second branch containing a short-circuit structure to obtain a fifth output element that is the same as the first input element, and the fourth output element and the fifth output element are weightedly added by an adder to obtain a sixth output element. This can achieve that when filtering processing is performed on at least one pixel to be filtered or the first filtering intermediate element according to the lookup table structure, filtering processing can be combined with the residual structure to reflect the advantages of the residual structure, thereby improving the filtering effect of the filtering processing.

[0404] Fifth embodiment

[0405] Based on any of the above embodiments, a fifth embodiment is proposed.

[0406] In this embodiment, the image processing method further includes at least one of the following methods 30 to 33:

[0407] Mode 30: In at least one intermediate element block, the spatial attention weights corresponding to at least two first intermediate elements are the same and / or the spatial attention weights corresponding to the first intermediate elements are different;

[0408] Optionally, for at least two first intermediate elements in at least one intermediate element block, spatial attention weights corresponding to the at least two first intermediate elements can be determined or obtained, and the spatial attention weights corresponding to the at least two first intermediate elements can be different.

[0409] Optionally, the method 30 may be combined with at least one of the methods 1 to 29 in the above embodiments. For example, the method 30 may be combined with the above method 9, wherein step S1 includes, for at least two first intermediate elements in at least one intermediate element block, determining or obtaining spatial attention weights corresponding to the at least two first intermediate elements based on at least one spatial feature and at least one lookup table, wherein the spatial attention weights corresponding to the at least two first intermediate elements may be different, and performing filtering processing on the at least one intermediate element block based on the spatial attention weights corresponding to the at least two first intermediate elements.

[0410] For example, if the size parameter of the intermediate element block is 3×3, the spatial attention weights corresponding to at least two of the nine first intermediate elements in the intermediate element block may be different. For example, the spatial attention weights corresponding to the nine first intermediate elements in the image block are all different.

[0411] Optionally, the spatial attention weights corresponding to the first intermediate elements in at least two rows or at least two columns of the intermediate element block may be different. For example, the spatial attention weight corresponding to the first intermediate element in the first row may be different from the spatial attention weight corresponding to the first intermediate element in the second row, or the spatial attention weights corresponding to the first intermediate elements in each row may be different.

[0412] Optionally, for at least two first intermediate elements in at least one intermediate element block, spatial attention weights corresponding to the at least two first intermediate elements can be determined or obtained, and the spatial attention weights corresponding to the at least two first intermediate elements can be the same.

[0413] For example, if the size parameter of the intermediate element block is 3×3, the spatial attention weights corresponding to at least two of the nine first intermediate elements in the intermediate element block may be the same. For example, the spatial attention weights corresponding to the nine first intermediate elements in the intermediate element block are all the same.

[0414] Optionally, the spatial attention weights corresponding to the first intermediate elements of at least two rows or at least two columns in the intermediate element block may be the same. For example, the spatial attention weight corresponding to the first intermediate element of the first row may be the same as the spatial attention weight corresponding to the first intermediate element of the second row, or the spatial attention weights corresponding to the first intermediate elements of each row may be the same.

[0415] Optionally, for at least three first intermediate elements in at least one intermediate element block, the spatial attention weights corresponding to the at least three first intermediate elements can be determined or obtained, the spatial attention weights corresponding to at least two first intermediate elements can be different, and the spatial attention weights corresponding to at least two first intermediate elements can be the same.

[0416] For example, if the size parameter of the intermediate element block is 3×3, the spatial attention weights corresponding to at least two of the nine first intermediate elements in the intermediate element block can be the same, and the spatial attention weights corresponding to at least two of the first intermediate elements can be different. For example, the spatial attention weights corresponding to four of the first intermediate elements in the intermediate element block are all the same, and the spatial attention weights corresponding to the other five first intermediate elements are different.

[0417] Optionally, the spatial attention weights corresponding to the first intermediate elements of at least two rows or at least two columns in the intermediate element block may be the same, and the spatial attention weights corresponding to the first intermediate elements of at least two rows or at least two columns may be different. For example, the spatial attention weight corresponding to the first intermediate element of the first row may be the same as the spatial attention weight corresponding to the first intermediate element of the second row, or the spatial attention weight corresponding to the first intermediate element of the second row may be different from the spatial attention weight corresponding to the first intermediate element of the third row.

[0418] Optionally, the spatial attention weights corresponding to at least two first intermediate elements in at least one intermediate element block can be determined or obtained based on at least one spatial feature of at least one intermediate element block and at least one lookup table, at least one activation function and at least one neural network. The spatial attention weights corresponding to the at least two first intermediate elements may be different, and the at least one intermediate element block is filtered based on the different spatial attention weights corresponding to the at least two first intermediate elements.

[0419] In this embodiment, for at least two first intermediate elements in at least one intermediate element block, spatial attention weights corresponding to the at least two first intermediate elements can be determined or obtained, and the spatial attention weights corresponding to the at least two first intermediate elements can be different and / or the same. The at least one intermediate element block is filtered based on the at least one spatial attention weight. The key image block features that need to be processed in the intermediate element block can be determined by different and / or the same spatial attention weights, and thus the key image block features can be focused during the filtering process, which can improve the filtering effect of the filtering process on the at least one intermediate element block.

[0420] Method 31: The spatial attention weight corresponding to at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to another first intermediate element;

[0421] Optionally, for at least two first intermediate elements in at least one intermediate element block, spatial attention weights corresponding to the at least two first intermediate elements can be determined or obtained, and the spatial attention weight corresponding to at least one first intermediate element is greater than the spatial attention weight corresponding to the other first intermediate element.

[0422] Optionally, Method 31 may be combined with at least one of Methods 1 to 29 in the above embodiments. For example, Method 31 is combined with Method 9, wherein step S1 includes, for at least two first intermediate elements in at least one intermediate element block, determining or obtaining spatial attention weights corresponding to the at least two first intermediate elements based on at least one spatial feature and at least one lookup table, wherein the spatial attention weight corresponding to the at least one first intermediate element is greater than the spatial attention weight corresponding to another first intermediate element, and filtering the at least one intermediate element block based on the spatial attention weights corresponding to the at least two first intermediate elements.

[0423] For example, if the size parameter of the intermediate element block is 3×3, the spatial attention weight corresponding to at least one of the nine first intermediate elements in the intermediate element block is greater than the spatial attention weight corresponding to another first intermediate element. For example, the spatial attention weight corresponding to one first intermediate element in the intermediate element block is greater than the spatial attention weights corresponding to the other eight first intermediate elements.

[0424] Optionally, the spatial attention weight corresponding to the first intermediate element of at least one row or column in the intermediate element block may be greater than the spatial attention weight corresponding to the first intermediate element of another row or column. For example, the spatial attention weight corresponding to the first intermediate element of the first row is greater than the spatial attention weight corresponding to the first intermediate element of the second row.

[0425] Optionally, based on at least one spatial feature and at least one lookup table, at least one activation function and at least one neural network, the spatial attention weights corresponding to at least two first intermediate elements in at least one intermediate element block can be determined or obtained, the spatial attention weight corresponding to at least one first intermediate element is greater than the spatial attention weight corresponding to another first intermediate element, and the at least one intermediate element block is filtered based on the spatial attention weights corresponding to the at least two first intermediate elements.

[0426] In this embodiment, for at least two first intermediate elements in at least one intermediate element block, spatial attention weights corresponding to the at least two first intermediate elements can be determined or obtained, and the spatial attention weight corresponding to at least one first intermediate element is greater than the spatial attention weight corresponding to the other first intermediate element. The at least one intermediate element block is filtered based on the at least one spatial attention weight. The key image block features that need to be processed in the intermediate element block can be determined by spatial attention weights of different sizes, and the key image block features can be focused during the filtering process, which can improve the filtering effect of the at least one intermediate element block.

[0427] Mode 32: In at least one intermediate element block, the spatial attention weight corresponding to the intermediate element region containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element region containing at least one other first intermediate element;

[0428] Optionally, the intermediate element block includes at least one intermediate element region, wherein the intermediate element region includes at least one first intermediate element. Optionally, an intermediate element region may correspond to at least one spatial attention weight, or at least one intermediate element region may correspond to one spatial attention weight.

[0429] Optionally, for at least two intermediate element regions in at least one intermediate element block, spatial attention weights corresponding to the at least two intermediate element regions can be determined or obtained, and the spatial attention weight corresponding to the intermediate element region containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element region containing at least one other first intermediate element.

[0430] Optionally, the thirty-second method can be combined with at least one of the methods one to twenty-nine in the above embodiments. For example, the thirty-second method is combined with the above method nine, and step S1 includes: for at least two intermediate element regions in at least one intermediate element block, spatial attention weights corresponding to the at least two intermediate element regions can be determined or obtained based on at least one spatial feature and at least one lookup table, wherein the spatial attention weight corresponding to the intermediate element region containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element region containing at least one other first intermediate element, and filtering processing is performed on the at least one intermediate element block based on the spatial attention weights corresponding to the at least two intermediate element regions.

[0431] For example, the size parameter of the intermediate element block is 3×3, and the intermediate element block contains two intermediate element areas, one intermediate element area contains the first intermediate elements of the first and second rows, and the other intermediate element area contains the first intermediate element of the third row. The spatial attention weight corresponding to the intermediate element area containing the first intermediate elements of the first and second rows can be greater than the spatial attention weight corresponding to the intermediate element area containing the first intermediate element of the third row.

[0432] Optionally, based on at least one spatial domain feature and at least one lookup table, at least one activation function and at least one neural network, the spatial attention weights corresponding to at least two intermediate element regions in at least one intermediate element block can be determined or obtained, the spatial attention weight corresponding to the intermediate element region containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element region containing at least one other first intermediate element, and the at least one intermediate element block is filtered based on the spatial attention weights corresponding to the at least two intermediate element regions.

[0433] In this embodiment, for at least two intermediate element regions in at least one intermediate element block, spatial attention weights corresponding to the at least two intermediate element regions can be determined or obtained, and the spatial attention weight corresponding to the intermediate element region containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element region containing at least one other first intermediate element. The at least one intermediate element block is filtered based on the at least one spatial attention weight, and the key image block features that need to be processed in the intermediate element block can be determined by spatial attention weights of different sizes, so that the key image block features can be focused during the filtering process, which can improve the filtering effect of the at least one intermediate element block.

[0434] Method 33: The spatial attention weight corresponding to the intermediate element sub-block containing at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to the intermediate element sub-block containing at least one other first intermediate element.

[0435] Optionally, the intermediate element block includes at least one intermediate element region, the intermediate element region includes at least one intermediate element sub-block, and the intermediate element sub-block includes at least one first intermediate element.

[0436] Optionally, an intermediate element sub-block may correspond to at least one spatial attention weight, or at least one intermediate element sub-block may correspond to a spatial attention weight.

[0437] Optionally, for at least two intermediate element sub-blocks in at least one intermediate element block, the spatial attention weights corresponding to the at least two intermediate element sub-blocks can be determined or obtained, and the spatial attention weight corresponding to the intermediate element sub-block containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element sub-block containing at least one other first intermediate element.

[0438] Optionally, Method 33 may be combined with at least one of Methods 1 to 29 in the above embodiments. For example, Method 33 is combined with Method 9 above, and step S1 includes: for at least two intermediate element sub-blocks in at least one intermediate element block, spatial attention weights corresponding to the at least two intermediate element sub-blocks may be determined or obtained based on at least one spatial domain feature and at least one lookup table, wherein the spatial attention weight corresponding to the intermediate element sub-block containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element sub-block containing at least one other first intermediate element, and filtering processing is performed on the at least one intermediate element block based on the spatial attention weights corresponding to the at least two intermediate element sub-blocks.

[0439] For example, the size parameter of the intermediate element block is 5×5. For a 3×3 intermediate element area contained in the intermediate element block, the intermediate element area contains two intermediate element sub-blocks, one intermediate element sub-block contains the first intermediate elements of the first row and the second row in the intermediate element area, and the other intermediate element sub-block contains the first intermediate element of the third row in the intermediate element area. The spatial attention weight corresponding to the intermediate element sub-block containing the first intermediate elements of the first row and the second row can be greater than the spatial attention weight corresponding to the intermediate element sub-block containing the first intermediate element of the third row.

[0440] Optionally, based on at least one spatial feature of at least one intermediate element block and at least one lookup table, at least one activation function and at least one neural network, the spatial attention weights corresponding to at least two intermediate element sub-blocks in at least one intermediate element block can be determined or obtained, the spatial attention weight corresponding to the intermediate element sub-block containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element sub-block containing at least one other first intermediate element, and the at least one intermediate element block is filtered based on the spatial attention weights corresponding to the at least two intermediate element sub-blocks.

[0441] In this embodiment, for at least two intermediate element sub-blocks in at least one intermediate element block, spatial attention weights corresponding to the at least two intermediate element sub-blocks can be determined or obtained, and the spatial attention weight corresponding to the intermediate element sub-block containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element sub-block containing at least one other first intermediate element. The at least one intermediate element block is filtered based on the at least one spatial attention weight, and the key image block features that need to be processed in the intermediate element block can be determined by spatial attention weights of different sizes, so that the key image block features can be focused during the filtering process, which can improve the filtering effect of the at least one intermediate element block.

[0442] Sixth embodiment

[0443] Based on any of the above embodiments, a sixth embodiment is proposed.

[0444] In this embodiment, filtering the intermediate element block corresponding to at least one image block according to at least one spatial attention weight includes at least one of the following methods 34 to 36:

[0445] Method 34: determining or obtaining at least one filter block or filtered intermediate element according to a result of performing spatial attention enhancement on a first intermediate element in at least one intermediate element block according to at least one spatial attention weight;

[0446] Optionally, step S1 includes determining or obtaining at least one spatial attention weight based on at least one spatial feature of at least one image block, determining or obtaining at least one filter block or filtered intermediate element based on the result of spatial attention enhancement of at least one first intermediate element based on the at least one spatial attention weight, and determining or obtaining at least one filtered image block based on the at least one filter block or filtered intermediate element.

[0447] Optionally, method 34 can be combined with at least one of methods 1 to 33 in the above embodiments. For example, method 34 is combined with method 30 in the above embodiment, and step S1 includes: determining or obtaining spatial attention weights corresponding to at least two first intermediate elements based on at least one spatial feature of at least one image block, the spatial attention weights corresponding to at least two first intermediate elements can be different, and determining or obtaining at least one filter block or filtered intermediate element based on the result of spatial attention enhancement of at least one first intermediate element based on the spatial attention weights corresponding to at least two first intermediate elements of at least one intermediate element block, and determining or obtaining at least one filtered image block based on the at least one filter block or filtered intermediate element.

[0448] Optionally, at least one spatial attention weight may be multiplied by at least one first intermediate element or an intermediate value obtained by processing at least one first intermediate element, and a result of spatial attention enhancement may be determined or obtained based on the result of the multiplication.

[0449] Optionally, at least one filter block may be determined or obtained based on the result of spatial attention enhancement, and at least one filtered image block may be determined or obtained based on the at least one filter block. For example, the at least one filtered image block includes the at least one filter block.

[0450] Optionally, at least one filtered intermediate element may be determined or obtained based on the result of spatial attention enhancement, and the at least one filtered intermediate element may be processed, for example, based on at least one of at least one neural network, at least one activation function, and at least one lookup table, and at least one filtered image block may be determined or obtained based on the result of the processing. For example, the at least one filtered image block may include the at least one processed filtered intermediate element.

[0451] Optionally, refer to Figure 7 , at least one spatial domain feature can be determined or obtained based on at least one image block, at least one spatial attention weight can be determined or obtained based on at least one spatial domain feature, at least one intermediate element block can be determined or obtained based on at least one image block and at least one lookup table, at least one filter block or filtered intermediate element can be determined or obtained based on the result of spatial attention enhancement of at least one intermediate element block, and at least one filtered image block can be determined or obtained based on the at least one filter block or filtered intermediate element.

[0452] In this embodiment, at least one spatial attention weight is determined or obtained based on at least one spatial feature of at least one image block, and at least one filter block or filtered intermediate element is determined or obtained based on the result of spatial attention enhancement of at least one first intermediate element according to the at least one spatial attention weight. At least one filtered image block is determined or obtained based on the at least one filter block or filtered intermediate element. The key image block features that need to be processed in the intermediate element block can be determined by the result of spatial attention enhancement of the first intermediate element according to the spatial attention weight, so that the key image block features can be focused during the filtering process, and the filtering effect of the filtering process on the at least one intermediate element block can be improved.

[0453] Method 35: Determine or obtain at least one filter block or filtered intermediate element based on the result of performing spatial attention enhancement on at least one intermediate element region in at least one intermediate element block according to at least one spatial attention weight;

[0454] Optionally, step S1 includes determining or obtaining at least one spatial attention weight based on at least one spatial feature of at least one image block, performing spatial attention enhancement on at least one intermediate element region in at least one intermediate element block based on the at least one spatial attention weight, determining or obtaining at least one filter block or filtered intermediate element, and determining or obtaining at least one filtered image block based on the at least one filter block or filtered intermediate element.

[0455] Optionally, method 35 can be combined with at least one of methods 1 to 33 in the above embodiments. For example, method 35 is combined with method 30 in the above embodiment, and step S1 includes: determining or obtaining spatial attention weights corresponding to at least two intermediate element regions based on at least one spatial domain feature of at least one image block, the spatial attention weights corresponding to at least two intermediate element regions may be different, and based on the spatial attention weights corresponding to at least two intermediate element regions of at least one intermediate element block, determining or obtaining at least one filter block or filtered intermediate element as a result of spatial attention enhancement of at least one intermediate element region in at least one intermediate element block, and determining or obtaining at least one filtered image block based on the at least one filter block or filtered intermediate element.

[0456] Optionally, at least one spatial attention weight may be multiplied by at least one intermediate element region or an intermediate value obtained by processing at least one intermediate element region, and a result of spatial attention enhancement may be determined or obtained based on the result of the multiplication.

[0457] Optionally, at least one filter block may be determined or obtained based on the result of spatial attention enhancement, and at least one filtered image block may be determined or obtained based on the at least one filter block. For example, the at least one filtered image block includes the at least one filter block.

[0458] Optionally, at least one filtered intermediate element may be determined or obtained based on the result of spatial attention enhancement, and the at least one filtered intermediate element may be processed, for example, based on at least one of at least one neural network, at least one activation function, and at least one lookup table, and at least one filtered image block may be determined or obtained based on the result of the processing. For example, the at least one filtered image block may include the at least one processed filtered intermediate element.

[0459] In this embodiment, at least one spatial attention weight is determined or obtained based on at least one spatial domain feature of at least one image block, and at least one filter block or filtered intermediate element is determined or obtained based on the result of spatial attention enhancement of at least one intermediate element region according to the at least one spatial attention weight. At least one filtered image block is determined or obtained based on the at least one filter block or filtered intermediate element. The key image block features that need to be processed in the intermediate element block can be determined by the result of spatial attention enhancement of the intermediate element region according to the spatial attention weight, so that the key image block features can be focused during the filtering process, and the filtering effect of the filtering process on the at least one intermediate element block can be improved.

[0460] Method thirty-six, based on at least one spatial attention weight, determines or obtains at least one filter block or filtered intermediate element based on the result of performing spatial attention enhancement on at least one intermediate element region in at least one intermediate element block.

[0461] Optionally, step S1 includes determining or obtaining at least one spatial attention weight based on at least one spatial feature of at least one image block, performing spatial attention enhancement on at least one intermediate element region in at least one intermediate element block based on the at least one spatial attention weight, determining or obtaining at least one filter block or filtered intermediate element, and determining or obtaining at least one filtered image block based on the at least one filter block or filtered intermediate element.

[0462] Optionally, method 36 can be combined with at least one of methods 1 to 33 in the above embodiments. For example, method 36 is combined with method 30 in the above embodiment, and step S1 includes: determining or obtaining spatial attention weights corresponding to at least two intermediate element regions based on at least one spatial domain feature of at least one image block, the spatial attention weights corresponding to at least two intermediate element regions can be different, and based on the spatial attention weights corresponding to at least two intermediate element regions of at least one intermediate element block, determining or obtaining at least one filter block or filtered intermediate element as a result of spatial attention enhancement for at least one intermediate element region in at least one intermediate element block, and determining or obtaining at least one filtered image block based on the at least one filter block or filtered intermediate element.

[0463] Optionally, at least one spatial attention weight may be multiplied by at least one intermediate element region or an intermediate value obtained by processing at least one intermediate element region, and a result of spatial attention enhancement may be determined or obtained based on the result of the multiplication.

[0464] Optionally, at least one filter block may be determined or obtained based on the result of spatial attention enhancement, and at least one filtered image block may be determined or obtained based on the at least one filter block. For example, the at least one filtered image block includes the at least one filter block.

[0465] Optionally, at least one filtered intermediate element may be determined or obtained based on the result of spatial attention enhancement, and the at least one filtered intermediate element may be processed, for example, based on at least one of at least one neural network, at least one activation function, and at least one lookup table, and at least one filtered image block may be determined or obtained based on the result of the processing. For example, the at least one filtered image block may include the at least one processed filtered intermediate element.

[0466] In this embodiment, at least one spatial attention weight is determined or obtained based on at least one spatial domain feature of at least one image block, and at least one filter block or filtered intermediate element is determined or obtained based on the result of spatial attention enhancement of at least one intermediate element region according to the at least one spatial attention weight. At least one filtered image block is determined or obtained based on the at least one filter block or filtered intermediate element. The key image block features that need to be processed in the intermediate element block can be determined by the result of spatial attention enhancement of the intermediate element region according to the spatial attention weight, so that the key image block features can be focused during the filtering process, and the filtering effect of the filtering process on the at least one intermediate element block can be improved.

[0467] Seventh embodiment

[0468] Based on any of the above embodiments, a seventh embodiment is proposed.

[0469] In this embodiment, the image processing method further includes determining or obtaining spatial features of at least one image block after image preprocessing.

[0470] Optionally, at least one spatial feature may be determined or obtained based on at least one image block after image preprocessing, and filtering may be performed on at least one image block or at least one image block after image preprocessing based on the at least one spatial feature.

[0471] Optionally, this embodiment can be combined with at least one of the methods 1 to 36 in the above embodiments.

[0472] In this embodiment, by determining or obtaining at least one spatial feature based on at least one image block after image preprocessing, and performing filtering processing on at least one image block or at least one image block after image preprocessing based on the at least one spatial feature, it is possible to comprehensively consider the spatial features of the at least one image block when filtering the at least one image block to determine key image block features that need to be processed in the image block, thereby focusing on the key image block features during filtering, and improving the filtering effect of the filtering processing on the at least one image block.

[0473] Optionally, the image preprocessing includes at least one of the following methods 37 to 40:

[0474] Method 37, image sharpening;

[0475] Optionally, a spatial feature of at least one image block after image sharpening may be determined or obtained, and filtering may be performed on the at least one image block or the at least one image block after image sharpening based on the at least one spatial feature.

[0476] Optionally, Method 37 may be combined with at least one of Methods 1 to 36 in the above embodiments. For example, Method 37 is combined with Method 1 in the above embodiments, where step S1 includes determining or obtaining a spatial feature of at least one image block after image sharpening, the spatial feature being determined or obtained based on a statistical feature value of a pixel to be filtered and at least one neighboring pixel in the at least one image block after image sharpening, and filtering the at least one image block or the at least one image block after image sharpening based on the at least one spatial feature.

[0477] Optionally, for video compression tasks, since higher frequency information such as texture is lost more, 3x3 USM sharpening (unsharp mask sharpening) can be used to enhance the texture intensity of the image block as a priori input. The image after USM sharpening is recorded as right Extract spatial features at each position (i, j), such as Figure 8 shown.

[0478] Optionally, for position (i, j), extract a 5×5 window around it. In order to make the most of the features of the entire window, divide the window into inner and outer regions, that is, perform the operation of the Mean module in the figure, and then average the two regions to obtain the spatial features. Optionally, calculate the feature mean of position (i, j) and its upper, lower, left, and right positions, that is, calculate the feature mean of position (i-1, j), position (i+1, j), position (i, j-1), and position (i, j+1), and record it as a inner, calculate the feature mean of the 8 outermost positions of the window, that is, calculate the feature mean of position (i-2,j-1), position (i-2,j+1), position (i-1,j-2), position (i+1,j-2), position (i+2,j-1), position (i+2,j+1), position (i-1,j+2), position (i+1,j+2), and record it as a exter ,(a inner , a exter ) is recorded as the spatial feature. According to the spatial feature of position (i, j), the spatial attention weight of position (i, j) can be further obtained:

[0479]

[0480] σ represents the sigmoid activation function; It can be a Conv1x2 layer or a lookup table. Conv1x2 indicates a 1x2 convolution kernel; style_feature is the spatial feature of the input, averaging the inner and outer features. In other words, to derive the spatial attention weight for position (i, j) based on the spatial features at position (i, j), a convolutional layer (Conv1x2) or a lookup table is used, followed by a sigmoid layer. Since the output of the sigmoid function ranges from 0 to 1, it can be used as an attention weight / mask. The lookup table can be obtained by storing the input and output of the convolutional layer (Conv1x2) in the lookup table.

[0481] Optionally, the result after spatial attention enhancement is:

[0482] f out =atten·f in +f in

[0483] f in It is the enhanced texture strength that serves as the prior input, and at the same time, the spatial attention weight is generated by spatial feature extraction, which further improves the network's ability to model spatial information and texture details.

[0484] In this embodiment, by determining or obtaining the spatial domain features of at least one image block after image sharpening, filtering is performed on at least one image block or at least one image block after image sharpening based on the at least one spatial domain feature. It is possible to avoid excessive loss of texture details through image sharpening. At the same time, when filtering at least one image block, the spatial domain features of at least one image block are comprehensively considered to determine the key image block features that need to be processed in the image block, and then the key image block features can be focused during filtering, which can improve the filtering effect of filtering at least one image block.

[0485] Method 38, gradient calculation;

[0486] Optionally, a spatial feature of at least one image block after gradient calculation may be determined or obtained, and filtering processing may be performed on the at least one image block or the at least one image block after gradient calculation based on the at least one spatial feature.

[0487] Optionally, Mode 38 may be combined with at least one of Modes 1 to 36 in the above embodiments. For example, Mode 38 is combined with Mode 1 in the above embodiments, where step S1 includes determining or obtaining a spatial feature of at least one image block after gradient calculation, the spatial feature being determined or obtained based on statistical feature values ​​of a pixel to be filtered and at least one neighboring pixel in the at least one image block after gradient calculation, and filtering the at least one image block or the at least one image block after image sharpening based on the at least one spatial feature.

[0488] In this embodiment, the spatial domain features of at least one image block after gradient calculation are determined or obtained, and filtering processing is performed on the at least one image block or the at least one image block after gradient calculation based on the at least one spatial domain feature. This allows for comprehensive consideration of the spatial domain features of the at least one image block when filtering the at least one image block to determine key image block features that need to be processed in the image block. This allows for focusing on the key image block features during filtering, thereby improving the filtering effect of the at least one image block.

[0489] Method 39, transformation;

[0490] Optionally, a spatial feature of at least one transformed image block may be determined or obtained, and filtering processing may be performed on the at least one image block or the at least one transformed image block according to the at least one spatial feature.

[0491] Optionally, the transform can be a Fourier transform, a wavelet transform, or a deep learning based feature transform network.

[0492] Optionally, the transformation may also be a scale transformation, such as image scaling (bilinear interpolation, bicubic interpolation), multi-scale pyramid construction (such as Gaussian pyramid, Laplacian pyramid).

[0493] Optionally, the transformation may also be a linear transformation, a nonlinear transformation, or a projective transformation, such as a Hough transform or a perspective transform.

[0494] Optionally, Method 39 may be combined with at least one of Methods 1 to 36 in the above embodiments. For example, Method 39 is combined with Method 1 in the above embodiments, where step S1 includes determining or obtaining a spatial feature of at least one transformed image block, the spatial feature being determined or obtained based on statistical feature values ​​of a pixel to be filtered and at least one neighboring pixel in the at least one transformed image block, and filtering the at least one image block or the at least one image block after image sharpening based on the at least one spatial feature.

[0495] In this embodiment, by determining or obtaining the spatial domain features of at least one transformed image block, filtering is performed on the at least one image block or the at least one transformed image block based on the at least one spatial domain feature. This allows for comprehensive consideration of the spatial domain features of the at least one image block when filtering the at least one image block to determine key image block features that need to be processed in the image block. This allows for focusing on the key image block features during filtering, thereby improving the filtering effect of the at least one image block.

[0496] Method 40: Perform texture detection based on a detection operator.

[0497] Optionally, a spatial feature of at least one image block after texture detection by a detection operator may be determined or obtained, and filtering may be performed on at least one image block or at least one image block after texture detection by a detection operator based on the at least one spatial feature.

[0498] Optionally, the detection operator includes at least one of a Sobel operator, a Scharr operator, a Canny operator, and a Laplacian operator.

[0499] Optionally, the detection operator may also be an operator based on a neural network.

[0500] Optionally, mode 40 may be combined with at least one of modes 1 to 36 in the above embodiments. For example, mode 40 is combined with mode 1 in the above embodiments, wherein step S1 includes determining or obtaining a spatial feature of at least one image block after texture detection is performed according to a detection operator, the spatial feature being determined or obtained based on a statistical feature value of a pixel to be filtered and at least one neighboring pixel in the at least one image block after texture detection is performed according to the detection operator, and filtering the at least one image block or the at least one image block after image sharpening based on the at least one spatial feature.

[0501] In this embodiment, by determining or obtaining the spatial domain features of at least one image block after texture detection is performed according to a detection operator, filtering is performed on at least one image block or at least one image block after texture detection is performed according to the detection operator based on the at least one spatial domain feature. This allows for comprehensive consideration of the spatial domain features of the at least one image block when filtering the at least one image block to determine key image block features that need to be processed in the image block, thereby focusing on the key image block features during filtering, and improving the filtering effect of filtering the at least one image block.

[0502] Eighth embodiment

[0503] The present application also provides a processing device, referring to Figure 9 , the processing device includes:

[0504] The processing module A1 performs filtering processing on at least one image block according to the spatial domain features of the at least one image block.

[0505] Optionally, the spatial domain feature is determined or obtained according to at least one of the following:

[0506] Statistical characteristic values ​​of a pixel to be filtered and at least one neighboring pixel in at least one image block;

[0507] Statistical characteristic values ​​of at least two non-neighbor pixels in at least one image block;

[0508] a statistical feature value of a pixel to be filtered and at least one of the following: a pixel to be filtered, and at least one upper neighboring pixel of the pixel to be filtered, at least one left neighboring pixel of the pixel to be filtered, at least one right neighboring pixel of the pixel to be filtered, and at least one lower neighboring pixel of the pixel to be filtered;

[0509] The weighted statistical characteristic values ​​of the pixel to be filtered and the pixels adjacent to the pixel to be filtered in at least one image block;

[0510] The statistical characteristic value of at least one pixel in the outermost layer of a window centered on the pixel to be filtered, wherein the size parameter of the window is less than or equal to the size parameter of the image block;

[0511] The statistical eigenvalues ​​of eight pixels that are not adjacent to the pixel to be filtered;

[0512] A statistical characteristic value of a sub-block to be filtered in at least one image block containing at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel;

[0513] The at least one image block includes a statistical characteristic value of at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel in the area to be filtered.

[0514] Optionally, the processing module A1 includes at least one of the following:

[0515] Determining or obtaining at least one spatial attention weight based on at least one spatial feature and at least one lookup table, and performing filtering processing on an intermediate element block corresponding to at least one image block based on the at least one spatial attention weight;

[0516] Determining or obtaining at least one spatial attention weight based on at least one spatial feature and at least one neural network, and performing filtering processing on an intermediate element block corresponding to at least one image block based on the at least one spatial attention weight;

[0517] At least one spatial attention weight is determined or obtained based on at least one spatial feature and at least one activation function, and filtering is performed on an intermediate element block corresponding to at least one image block based on the at least one spatial attention weight.

[0518] Optionally, the processing module A1 further includes at least one of the following:

[0519] Determining or obtaining at least one intermediate element block based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block;

[0520] The lookup table structure includes at least one of the following:

[0521] at least one lookup table;

[0522] at least one neural network module;

[0523] At least two search branches, at least one search branch having the same input as another search branch;

[0524] at least two search branches, the input of at least one search branch being determined or obtained based on the output of another search branch;

[0525] At least two search branches, the at least two search branches being located in at least one channel corresponding to at least one pixel to be filtered or a first filtered intermediate element;

[0526] At least two search branches, the search tables of the at least two search branches are searched in a parallel search and / or serial search manner;

[0527] at least two lookup tables, wherein the input of at least one lookup table is determined or obtained according to the type, size parameter and / or input range of the other lookup table;

[0528] At least two lookup tables, the at least two lookup tables being located in at least one channel corresponding to at least one pixel to be filtered or a first filtered intermediate element;

[0529] At least two lookup tables, wherein the lookup mode of the at least two lookup tables in the same channel is parallel search and / or serial search;

[0530] At least two lookup tables, wherein the lookup tables in at least two channels are searched in parallel and / or serially;

[0531] at least one search branch and at least one neural network module, wherein an input of the at least one neural network module is the same as an input of the at least one search branch;

[0532] at least one search branch and at least one neural network module, wherein an input of the at least one neural network module is determined or obtained based on an output of the at least one search branch;

[0533] At least one lookup table and at least one neural network module, wherein an input of the at least one neural network module is determined or obtained according to a type, size parameter and / or input range of the at least one lookup table;

[0534] at least two lookup tables, wherein the output ranges of the at least two lookup tables are the same and / or the output ranges of the lookup tables are different;

[0535] at least two lookup tables, at least two of which have the same input range and / or different input ranges;

[0536] At least one first residual module based on a lookup table, the first residual module comprising a first branch including at least one lookup table structure, an adder, and a second branch including a short-circuit structure;

[0537] At least one second residual module based on a lookup table, a first input element of the second residual module passes through a first branch containing at least one lookup table structure to obtain a fourth output element, and a first input element of the third residual module passes through a second branch containing a short-circuit structure to obtain a fifth output element that is the same as the first input element, and the fourth output element and the fifth output element are weightedly added by an adder to obtain a sixth output element.

[0538] Optionally, the processing module A1 further includes at least one of the following:

[0539] The spatial attention weights corresponding to at least two first intermediate elements in at least one intermediate element block are the same and / or the spatial attention weights corresponding to the first intermediate elements are different;

[0540] The spatial attention weight corresponding to at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to another first intermediate element;

[0541] The spatial attention weight corresponding to at least one intermediate element region containing at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to at least one intermediate element region containing at least one other first intermediate element;

[0542] The spatial attention weight corresponding to at least one intermediate element sub-block containing at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to at least one intermediate element sub-block containing at least one other first intermediate element.

[0543] Optionally, filtering the intermediate element block corresponding to the at least one image block according to the at least one spatial attention weight includes at least one of the following:

[0544] Determine or obtain at least one filter block or filtered intermediate element according to a result of performing spatial attention enhancement on a first intermediate element in at least one intermediate element block according to at least one spatial attention weight;

[0545] Determine or obtain at least one filter block or filtered intermediate element based on a result of performing spatial attention enhancement on at least one intermediate element region in at least one intermediate element block according to at least one spatial attention weight;

[0546] At least one filter block or filtered intermediate element is determined or obtained as a result of performing spatial attention enhancement on at least one intermediate element sub-block in at least one intermediate element block according to at least one spatial attention weight.

[0547] Optionally, the processing module A1 further includes: determining or obtaining spatial features of at least one image block after image preprocessing.

[0548] Optionally, the image preprocessing includes at least one of the following:

[0549] Image sharpening;

[0550] Gradient calculation;

[0551] Transformation;

[0552] Perform texture detection based on the detection operator.

[0553] The processing device provided in the embodiment of the present application has similar implementation principles and beneficial effects to the technical solutions shown in the above-mentioned corresponding method embodiments, and will not be described in detail here.

[0554] An embodiment of the present application further provides a processing device, including a memory and a processor. The memory stores an image processing program, and when the image processing program is executed by the processor, the steps of the image processing method in any of the above embodiments are implemented.

[0555] An embodiment of the present application further provides a storage medium on which an image processing program is stored. When the image processing program is executed by a processor, the steps of the image processing method in any of the above embodiments are implemented.

[0556] In the embodiments of the smart terminal and storage medium provided in this application, all technical features of any of the above-mentioned image processing method embodiments may be included. The expanded and explained contents of the specification are basically the same as those of the embodiments of the above-mentioned methods and will not be repeated here.

[0557] An embodiment of the present application further provides a computer program product, which includes computer program code. When the computer program code runs on a computer, the computer executes the methods in the various possible implementation modes described above.

[0558] An embodiment of the present application also provides a chip, including a memory and a processor, wherein the memory is used to store computer programs, and the processor is used to call and run the computer programs from the memory, so that a device equipped with the chip executes the methods in the various possible implementation modes as described above.

[0559] It is understood that the above scenarios are merely examples and do not limit the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, those skilled in the art will appreciate that with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application will also be applicable to similar technical problems.

[0560] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0561] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0562] The units in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0563] In this application, the same or similar terminology, technical solutions and / or application scenario descriptions are generally only described in detail the first time they appear. When they appear again later, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of this application, for the same or similar terminology, technical solutions and / or application scenario descriptions that are not described in detail later, you can refer to the previous relevant detailed descriptions.

[0564] In this application, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0565] The various technical features of the technical solution of this application can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0566] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as mentioned above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the method of each embodiment of the present application.

[0567] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a storage disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state storage disk Solid State Disk (SSD)).

[0568] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: Including steps: S1 , performing filtering processing on at least one image block according to a spatial domain feature of the at least one image block.

2. The image processing method according to claim 1, wherein: The airspace characteristics are determined or derived based on at least one of the following: Statistical characteristic values ​​of a pixel to be filtered and at least one neighboring pixel in at least one image block; Statistical characteristic values ​​of at least two non-neighbor pixels in at least one image block; a statistical feature value of a pixel to be filtered and at least one of the following: a pixel to be filtered, and at least one upper neighboring pixel of the pixel to be filtered, at least one left neighboring pixel of the pixel to be filtered, at least one right neighboring pixel of the pixel to be filtered, and at least one lower neighboring pixel of the pixel to be filtered; The weighted statistical characteristic values ​​of the pixel to be filtered and the pixels adjacent to the pixel to be filtered in at least one image block; The statistical characteristic value of at least one pixel in the outermost layer of a window centered on the pixel to be filtered, wherein the size parameter of the window is less than or equal to the size parameter of the image block; The statistical eigenvalues ​​of eight pixels that are not adjacent to the pixel to be filtered; A statistical characteristic value of a sub-block to be filtered in at least one image block containing at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel; The at least one image block includes a statistical characteristic value of at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel in the area to be filtered.

3. The image processing method according to claim 1, wherein: Step S1 includes at least one of the following: Determining or obtaining at least one spatial attention weight based on at least one spatial feature and at least one lookup table, and performing filtering processing on an intermediate element block corresponding to at least one image block based on the at least one spatial attention weight; Determining or obtaining at least one spatial attention weight based on at least one spatial feature and at least one neural network, and performing filtering processing on an intermediate element block corresponding to at least one image block based on the at least one spatial attention weight; At least one spatial attention weight is determined or obtained based on at least one spatial feature and at least one activation function, and filtering is performed on an intermediate element block corresponding to at least one image block based on the at least one spatial attention weight.

4. The image processing method according to claim 3, wherein: Also include at least one of the following: Determining or obtaining at least one intermediate element block based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block; The lookup table structure includes at least one of the following: at least one lookup table; at least one neural network module; At least two search branches, at least one search branch having the same input as another search branch; at least two search branches, the input of at least one search branch being determined or obtained based on the output of another search branch; At least two search branches, the at least two search branches being located in at least one channel corresponding to at least one pixel to be filtered or a first filtered intermediate element; At least two search branches, the search tables of the at least two search branches are searched in a parallel search and / or serial search manner; at least two lookup tables, wherein the input of at least one lookup table is determined or obtained according to the type, size parameter and / or input range of the other lookup table; At least two lookup tables, the at least two lookup tables being located in at least one channel corresponding to at least one pixel to be filtered or a first filtered intermediate element; At least two lookup tables, wherein the lookup mode of the at least two lookup tables in the same channel is parallel search and / or serial search; At least two lookup tables, wherein the lookup tables in at least two channels are searched in parallel and / or serially; at least one search branch and at least one neural network module, wherein an input of the at least one neural network module is the same as an input of the at least one search branch; at least one search branch and at least one neural network module, wherein an input of the at least one neural network module is determined or obtained based on an output of the at least one search branch; At least one lookup table and at least one neural network module, wherein an input of the at least one neural network module is determined or obtained according to a type, size parameter and / or input range of the at least one lookup table; at least two lookup tables, wherein the output ranges of the at least two lookup tables are the same and / or the output ranges of the lookup tables are different; at least two lookup tables, at least two of which have the same input range and / or different input ranges; At least one first residual module based on a lookup table, the first residual module comprising a first branch including at least one lookup table structure, an adder, and a second branch including a short-circuit structure; At least one second residual module based on a lookup table, a first input element of the second residual module passes through a first branch containing at least one lookup table structure to obtain a fourth output element, and a first input element of the third residual module passes through a second branch containing a short-circuit structure to obtain a fifth output element that is the same as the first input element, and the fourth output element and the fifth output element are weightedly added by an adder to obtain a sixth output element.

5. The image processing method according to claim 3, wherein: Also include at least one of the following: The spatial attention weights corresponding to at least two first intermediate elements in at least one intermediate element block are the same and / or the spatial attention weights corresponding to the first intermediate elements are different; The spatial attention weight corresponding to at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to another first intermediate element; The spatial attention weight corresponding to at least one intermediate element region containing at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to at least one intermediate element region containing at least one other first intermediate element; The spatial attention weight corresponding to at least one intermediate element sub-block containing at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to at least one intermediate element sub-block containing at least one other first intermediate element.

6. The image processing method according to claim 3, wherein: Performing filtering on an intermediate element block corresponding to at least one image block according to at least one spatial attention weight includes at least one of the following: Determine or obtain at least one filter block or filtered intermediate element according to a result of performing spatial attention enhancement on a first intermediate element in at least one intermediate element block according to at least one spatial attention weight; Determine or obtain at least one filter block or filtered intermediate element based on a result of performing spatial attention enhancement on at least one intermediate element region in at least one intermediate element block according to at least one spatial attention weight; At least one filter block or filtered intermediate element is determined or obtained based on a result of performing spatial attention enhancement on at least one intermediate element sub-block in at least one intermediate element block.

7. The image processing method according to any one of claims 1 to 6, wherein: Also includes: Determine or obtain the spatial domain features of at least one image block after image preprocessing.

8. The image processing method according to claim 7, wherein: Image preprocessing includes at least one of the following: Image sharpening; Gradient calculation; Transformation; Perform texture detection based on the detection operator.

9. A processing device, characterized in that include: A memory and a processor, wherein an image processing program is stored in the memory, and when the image processing program is executed by the processor, the steps of the image processing method according to any one of claims 1 to 8 are implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the image processing method according to any one of claims 1 to 8.

Images