Image processing method, processing device and storage medium

Abstract

The application provides an image processing method, a processing device and a storage medium. The image processing method comprises: performing filtering processing on at least one image block according to a spatial feature of the at least one image block. Through the technical scheme, the filtering effect of the filtering processing can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image processing method, processing device, and storage medium. Background Technology

[0002] Existing high-efficiency video coding frameworks, such as Neural Network Based Video Coding (NNVC) and / or Enhanced Compression Model (ECM), propose a video frame coding technique to improve coding performance without significantly increasing computational complexity.

[0003] In conceiving and implementing this application, the inventors discovered at least the following problem: During the loop filtering stage of the encoding / decoding process, for example, introducing a Low Complexity Neural Network Loop Filter (LC-NNLF) structure in NNVC for loop filtering processing. Because there are many image patch features, the neural network needs to process multiple image patch features, making it unable to focus on key image patch features, resulting in poor processing performance and affecting the filtering effect, which needs to be improved.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides an image processing method, processing device, and storage medium, aiming to solve the technical problem of how to improve the filtering effect of filtering processing.

[0006] This application provides an image processing method, applicable to a processing device, comprising the following steps:

[0007] S1, filter at least one image block based on the spatial characteristics of at least one image block.

[0008] Optionally, the spatial characteristics are determined or obtained based on at least one of the following:

[0009] Statistical feature values ​​of the pixel to be filtered and at least one neighboring pixel in at least one image block;

[0010] Statistical feature values ​​of at least two non-neighboring pixels in at least one image patch;

[0011] The statistical feature value of at least one of the following in at least one image block: the pixel to be filtered, the pixel above the pixel to be filtered, the pixel to the left of the pixel to be filtered, the pixel to the right of the pixel to be filtered, and the pixel below the pixel to be filtered;

[0012] The weighted statistical feature values ​​of the pixels to be filtered in at least one image block and the pixels adjacent to the pixels to be filtered;

[0013] The statistical feature value of at least one pixel in the outermost layer of the window centered on the pixel to be filtered, wherein the size parameter of the window is less than or equal to the size parameter of the image patch;

[0014] Statistical characteristic values ​​of eight pixels that are not adjacent to the pixel to be filtered;

[0015] At least one image block contains statistical feature values ​​of at least one of the following: at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel.

[0016] At least one image block contains statistical feature values ​​of the region to be filtered, including at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel.

[0017] Optionally, step S1 includes at least one of the following:

[0018] Based on at least one spatial feature and at least one lookup table, at least one spatial attention weight is determined or obtained, and the intermediate element block corresponding to at least one image block is filtered based on the at least one spatial attention weight.

[0019] Based on at least one spatial feature and at least one neural network, at least one spatial attention weight is determined or obtained, and the intermediate element block corresponding to at least one image block is filtered based on the at least one spatial attention weight.

[0020] Based on at least one spatial feature and at least one activation function, at least one spatial attention weight is determined or obtained, and the intermediate element block corresponding to at least one image block is filtered based on the at least one spatial attention weight.

[0021] Optionally, it may also include at least one of the following:

[0022] Based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block, determine or obtain at least one intermediate element block;

[0023] The lookup table structure includes at least one of the following:

[0024] At least one lookup table;

[0025] At least one neural network module;

[0026] There are at least two search branches, and at least one search branch has the same input as the other search branch;

[0027] There are at least two search branches, and the input of at least one search branch is determined or obtained based on the output of the other search branch;

[0028] At least two search branches, and at least two search branches are located in at least one channel corresponding to at least one pixel to be filtered or the first intermediate element of the filter;

[0029] There are at least two search branches, and the search method for the search table with at least two search branches is parallel search and / or serial search;

[0030] At least two lookup tables, and the input of at least one lookup table is determined or obtained based on the type, size parameters and / or input range of the other lookup table;

[0031] At least two lookup tables, and at least two lookup tables are located in at least one channel corresponding to at least one pixel to be filtered or the first intermediate element of the filter;

[0032] At least two lookup tables, and the lookup methods for at least two lookup tables within the same channel are parallel lookup and / or serial lookup;

[0033] At least two lookup tables, and the lookup tables in at least two channels are searched in parallel and / or serial manner;

[0034] At least one search branch and at least one neural network module, wherein the input of the at least one neural network module is the same as the input of the at least one search branch;

[0035] At least one search branch and at least one neural network module, wherein the input of the at least one neural network module is determined or obtained based on the output of the at least one search branch;

[0036] At least one lookup table and at least one neural network module, wherein the input of the at least one neural network module is determined or obtained based on the type, size parameters and / or input range of the at least one lookup table;

[0037] At least two lookup tables, at least two lookup tables have the same output range and / or the lookup tables have different output ranges;

[0038] At least two lookup tables, at least two lookup tables have the same input range and / or the lookup tables have different input ranges;

[0039] At least one first residual module based on a lookup table, the first residual module including a first branch containing at least one lookup table structure, an adder, and a second branch containing a short-circuit structure;

[0040] At least one second residual module based on a lookup table, the first input element of the second residual module is passed through a first branch containing at least one lookup table structure to obtain a fourth output element, and the first input element of the third residual module is passed through a second branch containing a short-circuit structure to obtain a fifth output element that is the same as the first input element. The fourth output element and the fifth output element are weighted and added by an adder to obtain a sixth output element.

[0041] Optionally, the image processing method further includes at least one of the following:

[0042] In at least one intermediate element block, at least two first intermediate elements have the same spatial attention weight and / or the first intermediate elements have different spatial attention weights.

[0043] In at least one intermediate element block, the spatial attention weight corresponding to at least one first intermediate element is greater than the spatial attention weight corresponding to the other first intermediate element.

[0044] The spatial attention weight corresponding to the region of at least one intermediate element that contains at least one first intermediate element is greater than the spatial attention weight corresponding to the region of at least one intermediate element that contains at least one other first intermediate element.

[0045] The spatial attention weight corresponding to the at least one intermediate element sub-block that contains at least one first intermediate element is greater than the spatial attention weight corresponding to the at least one intermediate element sub-block that contains at least one other first intermediate element.

[0046] Optionally, filtering is performed on the intermediate element block corresponding to at least one image patch based on at least one spatial attention weight, including at least one of the following:

[0047] Based on the spatial attention enhancement of the first intermediate element in at least one intermediate element block according to at least one spatial attention weight, at least one filter block or filter intermediate element is determined or obtained.

[0048] Based on at least one spatial attention weight, the spatial attention enhancement of at least one intermediate element region in at least one intermediate element block is performed to determine or obtain at least one filter block or filter intermediate element.

[0049] Based on at least one spatial attention weight, the spatial attention enhancement of at least one intermediate element sub-block in at least one intermediate element block is performed to determine or obtain at least one filter block or filter intermediate element.

[0050] Optionally, the image processing method further includes: determining or obtaining the spatial features of at least one image patch after image preprocessing.

[0051] Optionally, image preprocessing includes at least one of the following:

[0052] Image sharpening;

[0053] Gradient calculation;

[0054] Transformation;

[0055] Texture detection is performed based on the detection operator.

[0056] This application also provides a processing apparatus, including:

[0057] The processing module is used to perform filtering processing on at least one image block based on the spatial characteristics of at least one image block.

[0058] This application also provides a processing device, including: a memory and a processor, wherein the memory stores an image processing program, and when the image processing program is executed by the processor, it implements the steps of any of the image processing methods described above.

[0059] This application also provides a storage medium storing a computer program that, when executed by a processor, implements the steps of any of the image processing methods described above.

[0060] As described above, the image processing method of this application can be applied to a processing device, including: filtering at least one image block based on the spatial characteristics of at least one image block. Through the technical solution of this application, when filtering at least one image block, the spatial characteristics of at least one image block are comprehensively considered to determine the key image block features that need to be processed within the image block. This allows the filtering process to focus on the key image block features, thereby improving the filtering effect of filtering at least one image block. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0062] Figure 1 A schematic diagram of the hardware structure of a mobile terminal to implement the various embodiments of this application;

[0063] Figure 2 A communication network system architecture diagram provided for an embodiment of this application;

[0064] Figure 3 A schematic diagram of the hardware structure of a controller 140 provided in this application;

[0065] Figure 4 A schematic diagram of the hardware structure of a network node 150 provided in this application;

[0066] Figure 5 This is a flowchart illustrating the image processing method according to the first embodiment;

[0067] Figure 6 This is a flowchart illustrating the encoding and decoding process in image processing methods.

[0068] Figure 7 This is a flowchart of a filtering process in an image processing method;

[0069] Figure 8 This is a schematic diagram of spatial feature extraction of an image patch in an image processing method;

[0070] Figure 9 This is a schematic diagram of the processing module of the processing device.

[0071] The realization of the objectives, functional features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0072] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0073] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0074] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word “if” as used herein may be interpreted as “when…” or “in response to determination”. Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms “or,” “and / or,” “including at least one of the following,” etc., as used in this application may be interpreted as inclusive, or mean any one or any combination thereof. For example, "including at least one of the following: A, B, C" means "any one of the following: A; B; C; A and B; A and C; B and C; A and B and C." Similarly, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A and B and C." Exceptions to this definition only occur when the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0075] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0076] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0077] It should be noted that step designations such as S1 are used in this paper for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial restriction on the order.

[0078] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0079] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0080] The processing device in this application can be a smart terminal or a server. Optionally, the smart terminal can be implemented in various forms. For example, the smart terminal described in this application can include smart terminals such as mobile phones, tablets, laptops, handheld computers, personal digital assistants (PDAs), portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, etc., as well as fixed terminals such as digital TVs and desktop computers.

[0081] The following description will use a mobile terminal as an example. Those skilled in the art will understand that, apart from elements specifically designed for mobile purposes, the construction according to the embodiments of this application can also be applied to fixed-type terminals.

[0082] Please see Figure 1 This is a schematic diagram of the hardware structure of a mobile terminal implementing various embodiments of this application. The mobile terminal 100 may include: an RF (Radio Frequency) unit 101, a WiFi module 102, an audio output unit 103, an A / V (Audio / Video) input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, a processor 110, and a power supply 111, etc. Those skilled in the art will understand that... Figure 1 The mobile terminal structure shown does not constitute a limitation on the mobile terminal. The mobile terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0083] The following is combined with Figure 1 A detailed introduction to each component of the mobile terminal:

[0084] The radio frequency unit 101 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 110; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, and a duplexer. Furthermore, the radio frequency unit 101 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution), TDD-LTE (Time Division Duplexing-Long Term Evolution), 5G, and 6G.

[0085] WiFi is a short-range wireless transmission technology. Mobile terminals, through the WiFi module 102, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 1 WiFi module 102 is shown, but it is understood that it is not a necessary component of a mobile terminal and can be omitted as needed without changing the nature of the invention.

[0086] The audio output unit 103 can convert audio data received by the radio frequency unit 101 or the WiFi module 102 or stored in the memory 109 into audio signals and output them as sound when the mobile terminal 100 is in call signal receiving mode, call mode, recording mode, voice recognition mode, broadcast receiving mode, etc. Furthermore, the audio output unit 103 can also provide audio output related to specific functions performed by the mobile terminal 100 (e.g., call signal receiving sound, message receiving sound, etc.). The audio output unit 103 may include a speaker, a buzzer, etc.

[0087] The A / V input unit 104 is used to receive audio or video signals. The A / V input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on the display unit 106. The image frames processed by the GPU 1041 can be stored in the memory 109 (or other storage medium) or transmitted via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) in operating modes such as telephone call mode, recording mode, and voice recognition mode, and can process such sound into audio data. The processed audio (voice) data can be converted into a format that can be transmitted to a mobile communication base station via the radio frequency unit 101 in telephone call mode. The microphone 1042 can implement various types of noise cancellation (or suppression) algorithms to eliminate (or suppress) noise or interference generated during the reception and transmission of audio signals.

[0088] The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Optionally, the light sensor includes an ambient light sensor and a proximity sensor. Optionally, the ambient light sensor can adjust the brightness of the display panel 1061 according to the ambient light level, and the proximity sensor can turn off the display panel 1061 and / or backlight when the mobile terminal 100 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the phone, such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0089] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0090] User input unit 107 can be used to receive input numerical or character information, and generate key signal inputs related to user settings and function control of the mobile terminal. Optionally, user input unit 107 may include touch panel 1071 and other input devices 1072. Touch panel 1071, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 1071), and drive corresponding connection devices according to a pre-set program. Touch panel 1071 may include a touch detection device and a touch controller. Optionally, the touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to processor 110, and can receive and execute commands sent by processor 110. In addition, touch panel 1071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1071, the user input unit 107 may also include other input devices 1072. Optionally, other input devices 1072 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc., without being specifically limited here.

[0091] Optionally, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the processor 110 to determine the type of touch event. Subsequently, the processor 110 provides corresponding visual output on the display panel 1061 based on the type of touch event. Although in Figure 1 In this embodiment, the touch panel 1071 and the display panel 1061 are two independent components to realize the input and output functions of the mobile terminal. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize the input and output functions of the mobile terminal. The specific implementation is not limited here.

[0092] Interface unit 108 serves as an interface through which at least one external device can connect to mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 108 may be used to receive input (e.g., data, power, etc.) from the external device and transmit the received input to one or more elements within mobile terminal 100, or it may be used to transmit data between mobile terminal 100 and the external device.

[0093] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a program storage area and a data storage area. Optionally, the program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 109 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0094] The processor 110 is the control center of the mobile terminal. It connects various parts of the mobile terminal via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 109, and by calling data stored in the memory 109, it performs various functions and processes data of the mobile terminal, thereby providing overall monitoring of the mobile terminal. The processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor and a modem processor. Optionally, the application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 110.

[0095] The mobile terminal 100 may also include a power supply 111 (such as a battery) that supplies power to various components. Preferably, the power supply 111 can be logically connected to the processor 110 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0096] although Figure 1 As not shown, the mobile terminal 100 may also include a Bluetooth module, etc., which will not be described in detail here.

[0097] To facilitate understanding of the embodiments of this application, the communication network system on which the mobile terminal of this application is based is described below.

[0098] Please see Figure 2 , Figure 2 This application provides a communication network system architecture diagram. The communication network system is an LTE system based on the universal mobile communication technology. The LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203, and the operator's IP services 204, which are connected in sequence.

[0099] Optionally, UE201 can be the aforementioned terminal 100, which will not be described in detail here.

[0100] E-UTRAN202 includes eNodeB2021 and other eNodeB2022, etc. Optionally, eNodeB2021 can connect to other eNodeB2022 via backhaul (e.g., X2 interface), and eNodeB2021 connects to EPC203, providing access from UE201 to EPC203.

[0101] EPC203 may include MME (Mobility Management Entity) 2031, HSS (Home Subscriber Server) 2032, other MMEs 2033, SGW (Serving Gateway) 2034, PGW (Packet Data Network Gateway) 2035, and PCRF (Policy and Charging Rules Function) 2036, etc. Optionally, MME2031 is the control node that handles signaling between UE201 and EPC203, providing bearer and connection management. HSS2032 is used to provide registers to manage functions such as the Home Location Register (not shown in the figure) and stores user-specific information such as service characteristics and data rates. All user data can be sent through SGW2034. PGW2035 can provide UE 201 IP address allocation and other functions. PCRF2036 is the policy and charging control decision point for service data flow and IP bearer resources. It selects and provides available policy and charging control decisions for the policy and charging enforcement function unit (not shown in the figure).

[0102] IP services 204 may include the Internet, intranet, IMS (IP Multimedia Subsystem), or other IP services.

[0103] Although the above description uses the LTE system as an example, those skilled in the art should know that this application is not only applicable to the LTE system, but also to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA, 5G and future new network systems (such as 6G), etc., without limitation.

[0104] Figure 3This is a schematic diagram of the hardware structure of a controller 140 provided in this application. The controller 140 includes a memory 1401 and a processor 1402. The memory 1401 is used to store program instructions, and the processor 1402 is used to call the program instructions in the memory 1401 to execute the steps performed by the controller in the first embodiment of the above method. The implementation principle and beneficial effects are similar, and will not be described again here.

[0105] Optionally, the controller further includes a communication interface 1403, which can be connected to the processor 1402 via a bus 1404. The processor 1402 can control the communication interface 1403 to implement the receiving and sending functions of the controller 140.

[0106] Figure 4 This application provides a schematic diagram of the hardware structure of a network node 150. The network node 150 includes a memory 1501 and a processor 1502. The memory 1501 is used to store program instructions, and the processor 1502 is used to call the program instructions in the memory 1501 to execute the steps performed by the first node in the first embodiment of the above method. The implementation principle and beneficial effects are similar, and will not be described again here.

[0107] Optionally, the controller further includes a communication interface 1503, which can be connected to the processor 1502 via a bus 1504. The processor 1502 can control the communication interface 1503 to implement the receiving and sending functions of the network node 150.

[0108] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0109] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk, SSD), etc.

[0110] Based on the above-described mobile terminal hardware structure and communication network system, various embodiments of this application are proposed.

[0111] First Embodiment

[0112] Reference Figure 5 , Figure 5 This is a flowchart illustrating the image processing method according to the first embodiment. The image processing method of this application embodiment can be applied to a processing device, including step S10:

[0113] S10: Filter at least one image block based on its spatial characteristics.

[0114] In this embodiment, the processing device can be a smart terminal, such as a mobile phone or computer, or a server, such as a local server or a cloud server. This embodiment and this application primarily use a smart terminal as an example for illustration.

[0115] Optionally, the technical solution of this embodiment can be applied to fields such as image encoding and decoding, video encoding and decoding, hardware video encoding and decoding, dedicated circuit video encoding and decoding, and real-time video encoding and decoding.

[0116] Optionally, for ease of understanding, a brief introduction to the encoding and decoding process is provided: (e.g.) Figure 6As shown, the system includes modules such as general coding control, transform and quantization, intra-frame estimation, intra-frame prediction, motion compensation, motion estimation, inverse quantization and inverse transform, filter control analysis, deblocking filtering and SAO filtering (i.e., loop filtering), entropy coding, decoding frame buffer, and post-filtering (NNPF, Neural Network Post Filter). Optionally, the motion compensation module can perform intra-frame / inter-frame selection to determine the specific compensation. Optionally, during entropy coding, the coding bit rate is obtained based on the general control data determined by the general coding control module, the change quantization coefficients determined by the transform and quantization module, the intra-frame prediction data and filter control determined by the filter control analysis, and the motion data determined by the decoding frame buffer.

[0117] Optionally, the decoded video signal can be output through a decoding frame buffer.

[0118] Optionally, the loop filtering may include two branches: a deblocking filter and an LC-NNLF. The results of the branches are then fused and processed by SAO (Sample Adaptive Offset) and ALF (Adaptive Loop Filter).

[0119] Optionally, the technical solution of this embodiment can be applied to the filtering stage in the video encoding and / or decoding stage to perform filtering processing on at least one input image block. For example, it can be used in the loop filtering stage and / or the post-filtering stage.

[0120] Optionally, the image block can be an image block to be filtered, such as an image block to be filtered during the video encoding and / or decoding stage.

[0121] Optionally, the image block can be an image block obtained after preprocessing the image block to be filtered. For example, the preprocessing can be transformation, quantization, sharpening, etc., and there are no restrictions here.

[0122] Optionally, the image block may include at least one pixel to be filtered.

[0123] Optionally, the pixels to be filtered can be from an image block, or an image block obtained after preprocessing the image block to be filtered.

[0124] Optionally, the pixel to be filtered can be a predicted pixel (e.g., pixel prediction), a reconstructed pixel (e.g., pixel reconstruction), or a reference pixel.

[0125] Optionally, the predicted pixel, reconstructed pixel, and / or reference pixel can be an intra-frame pixel and / or an inter-frame pixel, or an intermediate value used or generated when the predicted pixel and / or reconstructed pixel generate the reference pixel, without limitation.

[0126] Optionally, the spatial features of an image patch may include at least one of the following: style features of the image patch, gradient distribution, edge orientation histogram, color component correlation, and motion activity.

[0127] Optionally, gradient distributions such as those for VVC (Versatile Video Coding, a next-generation video coding standard jointly developed by the ITU Telecommunication Standardization Sector and the International Organization for Standardization / International Electrotechnical Commission Moving Picture Experts Group) and ALF (Adaptive Loop Filter) gradients can be calculated.

[0128] Optionally, the edge orientation histogram corresponds to AV1 (AOMedia Video 1, an open-source, royalty-free next-generation video coding standard developed by the Open Media Consortium) directional filtering.

[0129] Optionally, color component correlation covers VVC CCLM (Cross-Component Linear Model) chromaticity prediction.

[0130] Optionally, activity level is used for dynamic QP (quantization parameter) adjustment.

[0131] Optionally, one image patch may correspond to one spatial feature, or two or more image patches may correspond to one spatial feature, or one image patch may correspond to two or more spatial features, or two or more image patches may correspond to two or more spatial features.

[0132] Optionally, each pixel to be filtered in an image block may correspond to at least one spatial feature.

[0133] Optionally, a spatial feature can be determined or obtained based on at least one pixel to be filtered in at least one image block.

[0134] Optionally, the spatial features of an image patch may include at least one spatial feature determined or obtained based on at least one pixel to be filtered in the image patch.

[0135] Optionally, a spatial feature can be determined or obtained based on a pixel to be filtered, or two or more spatial features can be determined or obtained based on a pixel to be filtered, or a spatial feature can be determined or obtained based on two or more pixels to be filtered, or two or more spatial features can be determined or obtained based on two or more pixels to be filtered.

[0136] Optionally, the spatial features of an image patch may include the spatial features corresponding to all or part of the pixels to be filtered in the image patch. The spatial features corresponding to a pixel to be filtered may be determined or obtained based on at least one pixel to be filtered in the image patch, such as based on the pixel to be filtered and other pixels to be filtered in the image patch.

[0137] Optionally, the spatial characteristics of at least one image block can be determined or obtained based on the statistical characteristic value of at least one pixel to be filtered in the at least one image block.

[0138] Optionally, the spatial characteristics of at least one image block can be determined or obtained by preprocessing the image block and then determining or obtaining them based on the statistical feature value of at least one pixel / element to be filtered in the preprocessed image block.

[0139] Optionally, statistical feature values ​​may include at least one of the following: mean, maximum, minimum, median, mode (the pixel value that appears most frequently), range (the difference between the maximum and minimum values), variance, standard deviation, etc., or may include values ​​determined or obtained according to other rules or calculation methods, without limitation.

[0140] Optionally, the spatial characteristics of an image patch can be determined or obtained by sequentially calculating the statistical feature values ​​of all pixels to be filtered in the image patch, and then using these statistical feature values.

[0141] Optionally, an image patch can be filtered based on its spatial characteristics, or it can be filtered based on the spatial characteristics of two or more image patches. For example, an image patch can be filtered based on its spatial characteristics.

[0142] Optionally, an image patch may contain image patch features from at least one channel. Optionally, the image patch features from at least one channel may be filtered based on the spatial features of an image patch.

[0143] Optionally, the processing device can be a decoding end, and if at the decoding end, the image block can be a decoded image block.

[0144] Optionally, the processing device can be an encoding end, and if at the encoding end, the image block can be a filtered image block.

[0145] In this embodiment, by filtering at least one image block based on its spatial features, the spatial features of at least one image block are comprehensively considered during the filtering process to determine the key image block features that need to be processed. This allows the filtering process to focus on the key image block features and improves the filtering effect of filtering at least one image block.

[0146] Second Embodiment

[0147] Based on the first embodiment, a second embodiment is proposed.

[0148] In this embodiment, the spatial characteristics are determined or obtained according to at least one of the following methods one to eight:

[0149] Method 1: Statistical feature values ​​of the pixel to be filtered and at least one neighboring pixel in at least one image block;

[0150] Optionally, step S1 includes: filtering at least one image block based on the spatial features of at least one image block, wherein the spatial features are determined or obtained based on the statistical feature values ​​of the pixel to be filtered and at least one neighboring pixel in the at least one image block.

[0151] Optionally, the statistical feature values ​​may include at least one of the following: mean, maximum, minimum, median, mode (the pixel value that appears most frequently), range (the difference between the maximum and minimum values), variance, standard deviation, etc., of the pixel to be filtered and at least one neighboring pixel in at least one image block; or they may include values ​​determined or obtained according to other rules or calculation methods, without limitation.

[0152] Optionally, neighboring pixels can be pixels within a certain range around the pixel to be filtered in the image block. For example, neighboring pixels can include pixels within a preset distance from the pixel to be filtered in at least one of the eight directions: directly to the left, directly above, directly to the right, directly below, upper left, upper right, lower left, and lower right of the pixel to be filtered.

[0153] Optionally, the preset distance can be determined based on the number of pixels contained in the image patch. For example, the more pixels in the image patch, the larger the preset distance can be set. Optionally, the preset distance can be set to 1 or 2.

[0154] Optionally, neighboring pixels may include pixels in the image block that are directly adjacent to the pixel to be filtered, or may include pixels in the image block that are indirectly adjacent to the pixel to be filtered.

[0155] Optionally, for example, neighboring pixels may include eight pixels in the image block that are directly to the left, above, right, below, upper left, upper right, lower left, and lower right of the pixel to be filtered, or may include 16 pixels surrounding the eight pixels.

[0156] Optionally, the spatial features of at least one image block can be determined or obtained based on the average value of the pixel to be filtered and at least one neighboring pixel in at least one image block, and the at least one image block can be filtered based on the spatial features of at least one image block.

[0157] Optionally, the spatial features of at least one image block can be determined or obtained based on the average value of the pixel to be filtered and eight pixels directly to the left, above, right, below, upper left, upper right, lower left, and lower right of the pixel to be filtered, and the at least one image block can be filtered based on the spatial features of the at least one image block.

[0158] Optionally, the spatial features of at least one image block can be determined or obtained based on the statistical feature values ​​of the pixel to be filtered and at least one neighboring pixel of the pixel to be filtered in the at least one image block, and the at least one image block can be filtered based on the spatial features of the at least one image block.

[0159] Optionally, for all or part of the pixels to be filtered in at least one image block, at least one neighboring pixel of the pixel to be filtered in at least one image block can be determined, statistical feature values ​​of the pixel to be filtered and at least one neighboring pixel can be determined, and at least one spatial feature of at least one image block can be determined or obtained based on the statistical feature values.

[0160] In this embodiment, by filtering at least one image block based on the spatial features of at least one image block, it is possible to comprehensively consider the spatial features determined or obtained based on the statistical feature values ​​of the pixel to be filtered and at least one neighboring pixel in the at least one image block during the filtering process. This helps to identify the key image block features that need to be processed in the image block, thereby focusing on the key image block features during the filtering process and improving the filtering effect of filtering at least one image block.

[0161] Method 2: Statistical feature values ​​of at least two non-neighboring pixels in at least one image block;

[0162] Optionally, step S1 includes: filtering at least one image block based on the spatial features of at least one image block, wherein the spatial features are determined or obtained based on the statistical feature values ​​of at least two non-neighboring pixels in the at least one image block.

[0163] Optionally, the statistical feature value may include at least one of the following: mean, maximum, minimum, median, mode (the pixel value that appears most frequently), range (the difference between the maximum and minimum values), variance, standard deviation, etc., of at least two non-neighboring pixels in at least one image patch, or may include values ​​determined or obtained according to other rules or calculation methods, without limitation.

[0164] Optionally, non-neighbor pixels can be pixels in the image block other than the pixel to be filtered and neighboring pixels. For example, non-neighbor pixels can be pixels in the image block that are outside a certain range around the pixel to be filtered. For example, non-neighbor pixels can include pixels that are more than a preset distance away from the pixel to be filtered in at least one of the eight directions: directly to the left, directly above, directly to the right, directly below, upper left, upper right, lower left, and lower right of the pixel to be filtered.

[0165] Optionally, the preset distance can be determined based on the number of pixels contained in the image patch. For example, the more pixels in the image patch, the larger the preset distance can be set. Optionally, the preset distance can be set to 1 or 2.

[0166] For example, if the size of the image patch is 7×7, for the pixel to be filtered in the middle, the non-neighbor pixels can include at least one of the 24 outermost pixels of the image patch.

[0167] Optionally, the spatial features of at least one image block can be determined or obtained based on the statistical feature values ​​of at least two non-neighboring pixels in the at least one image block, and the at least one image block can be filtered based on the spatial features of the at least one image block.

[0168] Optionally, for all or part of the pixels to be filtered in at least one image block, at least two non-neighbor pixels of the pixel to be filtered in at least one image block can be determined, statistical feature values ​​of the at least two non-neighbor pixels can be determined, and at least one spatial feature of at least one image block can be determined or obtained based on the statistical feature values.

[0169] In this embodiment, by filtering at least one image block based on the spatial features of at least one image block, and comprehensively considering the spatial features determined or obtained based on the statistical feature values ​​of at least two non-neighboring pixels in at least one image block, the key image block features that need to be processed in the image block are determined, thereby focusing on the key image block features during the filtering process, which can improve the filtering effect of filtering at least one image block.

[0170] Method 3: The statistical feature value of at least one of the following in at least one image block: the pixel to be filtered, the pixel above the pixel to be filtered, the pixel to the left of the pixel to be filtered, the pixel to the right of the pixel to be filtered, and the pixel below the pixel to be filtered.

[0171] Optionally, step S1 includes: performing filtering processing on at least one image block based on the spatial domain features of at least one image block, wherein the spatial domain features are determined or obtained based on the statistical feature value of at least one of the following: the pixel to be filtered in the at least one image block, at least one upper neighbor pixel, at least one left neighbor pixel, at least one right neighbor pixel, and at least one lower neighbor pixel.

[0172] Optionally, the statistical feature values ​​may include at least one of the following: mean, maximum, minimum, median, mode (the pixel value with the highest frequency), range (the difference between the maximum and minimum values), variance, standard deviation, etc., of at least one of the following: the pixel to be filtered, at least one of the following: the pixel above the pixel to be filtered, the pixel to the left of the pixel to be filtered, the pixel to the right of the pixel to be filtered, and the pixel to the bottom of the pixel to be filtered. Alternatively, the statistical feature values ​​may include values ​​determined or obtained according to other rules or calculation methods, without limitation.

[0173] Optionally, the adjacent pixel can be a pixel in the image block that is adjacent to the pixel to be filtered. The adjacent pixels of the pixel to be filtered can include at least one of the following: at least one above adjacent pixel, at least one below adjacent pixel, at least one right adjacent pixel, and at least one below adjacent pixel.

[0174] Optionally, the spatial features of at least one image block can be determined or obtained based on the statistical feature value of at least one of the following: the pixel to be filtered in at least one image block, the pixel to be filtered above, the pixel to be filtered to the left, the pixel to be filtered to the right, and the pixel to be filtered to the right below. The at least one image block can then be filtered based on the spatial features of the at least one image block.

[0175] Optionally, for all or part of the pixels to be filtered in at least one image block, the statistical feature value of the pixel to be filtered and at least one of the following: at least one upper neighboring pixel, at least one left neighboring pixel, at least one right neighboring pixel, and at least one lower neighboring pixel can be determined, and at least one spatial feature of the at least one image block can be determined or obtained based on the statistical feature value.

[0176] Optionally, the spatial features of at least one image block can be determined or obtained based on the statistical feature values ​​of the pixel to be filtered in at least one image block and its adjacent pixels above, to the left, to the right, and to the bottom, and the at least one image block can be filtered based on the spatial features of the at least one image block.

[0177] In this embodiment, filtering is performed on at least one image block based on its spatial characteristics. The key image block features that need to be processed in the image block are determined by comprehensively considering the spatial characteristics determined or obtained by the statistical feature values ​​of at least one of the following: the pixel to be filtered, the pixel above the pixel to be filtered, the pixel to the left of the pixel to be filtered, the pixel to the right of the pixel to be filtered, and the pixel to the bottom of the pixel to be filtered. This allows the filtering effect of the filtering process on the at least one image block to be improved by focusing on the key image block features during the filtering process.

[0178] Method 4: The weighted statistical feature values ​​of the pixels to be filtered and their adjacent pixels in at least one image block;

[0179] Optionally, step S1 includes: filtering at least one image block based on the spatial features of at least one image block, wherein the spatial features are determined or obtained based on the weighted statistical feature values ​​of the pixel to be filtered and the pixels adjacent to the pixel to be filtered in the at least one image block.

[0180] Alternatively, the concept of adjacent pixels can be understood using the method described above (method 3).

[0181] Optionally, the weighted statistical feature value may include at least one of the following: the weighted mean, weighted maximum value, weighted minimum value, weighted median, weighted mode (the pixel value that appears most frequently), weighted range (the difference between the maximum and minimum values), weighted variance, weighted standard deviation, etc., of the pixel to be filtered in at least one image block; or it may include values ​​determined or obtained according to other rules or calculation methods, without limitation.

[0182] Optionally, the spatial features of at least one image block can be determined or obtained based on the weighted statistical feature value of at least one of the following: the pixel to be filtered in at least one image block, the pixel to be filtered above, the pixel to be filtered to the left, the pixel to be filtered to the right, and the pixel to be filtered to the right below. The at least one image block can then be filtered based on the spatial features of the at least one image block.

[0183] Optionally, the weights of the weighting can be preset.

[0184] Optionally, the weights can be obtained through pre-training / offline training of the neural network.

[0185] Optionally, the weights of different pixels can be the same or different.

[0186] Optionally, for all or part of the pixels to be filtered in at least one image block, a weighted statistical feature value can be determined for the pixel to be filtered and at least one of at least one of the following: the pixel to be filtered above, the pixel to be filtered to the left, the pixel to be filtered to the right, and the pixel to be filtered to the right below. At least one spatial feature of the at least one image block can be determined or obtained based on the weighted statistical feature value.

[0187] Optionally, the spatial features of at least one image block can be determined or obtained based on the weighted statistical feature values ​​of the pixel to be filtered in at least one image block and its adjacent pixels above, to the left, to the right, and to the bottom, and the at least one image block can be filtered based on the spatial features of the at least one image block.

[0188] In this embodiment, by filtering at least one image block based on its spatial features, and by comprehensively considering the spatial features determined or obtained from the weighted statistical feature values ​​of the pixels to be filtered and the pixels adjacent to the pixels to be filtered in the at least one image block, the key image block features that need to be processed in the image block are determined. Thus, the key image block features can be focused on during the filtering process, which can improve the filtering effect of filtering at least one image block.

[0189] Method 5: Statistical feature value of at least one pixel in the outermost layer of the window centered on the pixel to be filtered, where the window size parameter is less than or equal to the image patch size parameter;

[0190] Optionally, step S1 includes: filtering at least one image block based on the spatial features of at least one image block, wherein the spatial features are determined or obtained based on the statistical feature value of at least one pixel at the outermost layer of a window centered on the pixel to be filtered, and the size parameter of the window is less than or equal to the size parameter of the image block.

[0191] Optionally, the statistical feature values ​​may include at least one of the following: mean, maximum, minimum, median, mode (the pixel value that appears most frequently), range (the difference between the maximum and minimum values), variance, standard deviation, etc., of at least one pixel in the outermost layer of the window centered on the pixel to be filtered; or they may include values ​​determined or obtained according to other rules or calculation methods, without limitation.

[0192] Optionally, the window can be a window centered on the pixel to be filtered, and the window size parameter can be set to be less than or equal to the image block size parameter.

[0193] Optionally, the spatial features of at least one image block can be determined or obtained by using the statistical feature value of at least one pixel at the outermost layer of the window centered on the pixel to be filtered in at least one image block, and the at least one image block can be filtered based on the spatial features of the at least one image block.

[0194] Optionally, for all or part of the pixels to be filtered in at least one image block, the statistical feature value of at least one pixel at the outermost layer of the window centered on the pixel to be filtered in the at least one image block can be determined, and at least one spatial feature of the at least one image block can be determined or obtained based on the statistical feature value.

[0195] For example, if the size parameter of the image block is 7×7 and the size parameter of the window is set to 5×5, the spatial characteristics of at least one image block can be determined or obtained based on the statistical feature value of at least one pixel among the 16 pixels in the outermost layer of the 5×5 window centered on the pixel to be filtered.

[0196] Optionally, the spatial features of at least one image block can be determined or obtained by using the weighted statistical feature value of at least one pixel at the outermost layer of the window centered on the pixel to be filtered in at least one image block, and the at least one image block can be filtered based on the spatial features of the at least one image block.

[0197] In this embodiment, by filtering at least one image block based on the spatial features of at least one image block, and comprehensively considering the spatial features determined or obtained based on the statistical feature value of at least one pixel at the outermost layer of the window centered on the pixel to be filtered, the key image block features that need to be processed in the image block are determined, thereby focusing on the key image block features during the filtering process, which can improve the filtering effect of filtering at least one image block.

[0198] Method 6: Statistical feature values ​​of eight pixels that are not adjacent to the pixel to be filtered;

[0199] Optionally, step S1 includes: filtering at least one image block based on the spatial features of at least one image block, wherein the spatial features are determined or obtained based on the statistical feature values ​​of eight pixels that are not adjacent to the pixel to be filtered.

[0200] Optionally, the statistical feature values ​​may include at least one of the following: mean, maximum, minimum, median, mode (the pixel value that appears most frequently), range (the difference between the maximum and minimum values), variance, standard deviation, etc., of the eight pixels that are not adjacent to the pixel to be filtered, or may include values ​​determined or obtained according to other rules or calculation methods, without limitation.

[0201] Optionally, eight pixels can be determined from the pixels that are not adjacent to the pixel to be filtered in at least one image block. Based on the statistical feature values ​​of the determined eight pixels, the spatial features of at least one image block can be determined or obtained, and the at least one image block can be filtered based on the spatial features of at least one image block.

[0202] Optionally, for all or part of the pixels to be filtered in at least one image block, statistical feature values ​​that are not adjacent to the pixels to be filtered in at least one image block can be determined, and at least one spatial feature of at least one image block can be determined or obtained based on the statistical feature values.

[0203] Optionally, for example, if the size of the image block is 5×5, for the middle pixel to be filtered in the image block, 8 pixels can be determined from 16 pixels that are not adjacent to the pixel to be filtered. Based on the statistical feature values ​​of the determined 8 pixels, at least one spatial feature of at least one image block can be determined or obtained.

[0204] Optionally, eight pixels can be determined from the pixels that are not adjacent to the pixel to be filtered in at least one image block. Based on the weighted statistical feature values ​​of the eight determined pixels, the spatial features of at least one image block can be determined or obtained. The at least one image block is then filtered based on the spatial features of the at least one image block. The weighting weights of each pixel can be the same or different.

[0205] Optionally, the eight pixels that are not adjacent to the pixel to be filtered can be mutually non-adjacent. For example, if the size of the image block is 5×5, for the pixel to be filtered in the middle of the image block, eight non-adjacent pixels can be determined from the 16 non-adjacent pixels to the pixel to be filtered. Based on the statistical feature values ​​of the eight non-adjacent pixels, at least one spatial feature of at least one image block can be determined or obtained.

[0206] In this embodiment, by filtering at least one image block based on the spatial characteristics of at least one image block, and comprehensively considering the spatial characteristics determined or obtained based on the statistical feature values ​​of eight pixels that are not adjacent to the pixel to be filtered, the key image block features that need to be processed in the image block are determined, and thus the key image block features can be focused on during the filtering process, which can improve the filtering effect of filtering at least one image block.

[0207] Method 7: At least one image block contains the statistical feature value of at least one of the following: at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel.

[0208] Optionally, step S1 includes: filtering at least one image block based on the spatial features of at least one image block, wherein the spatial features are determined or obtained based on the statistical feature values ​​of the sub-block to be filtered, which contains at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel.

[0209] Optionally, the statistical feature values ​​may include at least one of the following: mean, maximum, minimum, median, mode (the pixel value that appears most frequently), range (the difference between the maximum and minimum values), variance, standard deviation, etc., of a sub-block to be filtered that contains at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel in at least one image block. Alternatively, they may include values ​​determined or obtained according to other rules or calculation methods, without limitation.

[0210] Optionally, the concept of neighboring pixels can refer to Method 1 above, and the concept of non-neighboring pixels can refer to Method 2 above.

[0211] Optionally, an image block may include at least one sub-block to be filtered, which may include at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel.

[0212] Optionally, each sub-block to be filtered in an image block may correspond to at least one spatial feature.

[0213] Optionally, a spatial feature of an image patch can be determined or obtained based on at least one sub-block to be filtered in at least one image patch. Optionally, the spatial feature of an image patch may include at least one spatial feature determined or obtained based on at least one sub-block to be filtered in the image patch.

[0214] Optionally, the sub-block to be filtered may include at least one pixel to be filtered and at least one neighboring pixel of the pixel to be filtered; or, the sub-block to be filtered may include at least one pixel to be filtered and at least one non-neighboring pixel of the pixel to be filtered; or, the sub-block to be filtered may include at least one pixel to be filtered, at least one neighboring pixel of the pixel to be filtered, and at least one non-neighboring pixel of the pixel to be filtered.

[0215] Optionally, a sub-block to be filtered can be determined from the image block, which includes at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel. The spatial features of at least one image block can be determined or obtained based on the statistical feature values ​​of the sub-block to be filtered, and the at least one image block can be filtered based on the spatial features of the at least one image block.

[0216] Optionally, for all or part of the pixels to be filtered in at least one image block, a sub-block to be filtered can be determined, which includes at least one of the pixels to be filtered, at least one neighboring pixel of the pixels to be filtered, and at least one non-neighboring pixel of the pixels to be filtered. The spatial features of the at least one image block can be determined or obtained based on the statistical feature value of the sub-block to be filtered, and the at least one image block can be filtered based on the spatial features of the at least one image block.

[0217] Optionally, a sub-block to be filtered can be determined from the image block, which includes at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel. The spatial features of at least one image block can be determined or obtained based on the weighted statistical feature value of the sub-block to be filtered, and the at least one image block can be filtered based on the spatial features of the at least one image block.

[0218] In this embodiment, by filtering at least one image block based on the spatial features of at least one image block, and comprehensively considering the spatial features determined or obtained based on the statistical feature values ​​of the sub-block to be filtered, which includes at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel, the key image block features that need to be processed in the image block are determined. Thus, the key image block features can be focused on during the filtering process, which can improve the filtering effect of filtering at least one image block.

[0219] Method 8: At least one image block contains statistical feature values ​​of the region to be filtered, including at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel.

[0220] Optionally, step S1 includes: filtering at least one image block based on the spatial features of at least one image block, wherein the spatial features are determined or obtained based on the statistical feature values ​​of the region to be filtered that contains at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel in the at least one image block.

[0221] Optionally, the statistical feature values ​​may include at least one of the following: mean, maximum, minimum, median, mode (the pixel value that appears most frequently), range (the difference between the maximum and minimum values), variance, standard deviation, etc., of the region to be filtered that contains at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel in at least one image patch. Alternatively, they may include values ​​determined or obtained according to other rules or calculation methods, without limitation.

[0222] Optionally, the concept of neighboring pixels can refer to Method 1 above, and the concept of non-neighboring pixels can refer to Method 2 above.

[0223] Optionally, an image block may include at least one region to be filtered, and the region to be filtered may include at least one sub-block to be filtered. The concept of the sub-block to be filtered can be referred to in Method 7 above.

[0224] Optionally, each region to be filtered in an image patch may correspond to at least one spatial feature.

[0225] Optionally, a spatial feature of an image patch can be determined or obtained based on at least one region to be filtered in at least one image patch.

[0226] Optionally, the spatial features of an image patch may include at least one spatial feature determined or obtained based on at least one region to be filtered in the image patch.

[0227] Optionally, a region to be filtered can be determined from an image block, which includes at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel. The spatial features of at least one image block can be determined or obtained based on the statistical feature values ​​of the region to be filtered, and the at least one image block can be filtered based on the spatial features of the at least one image block.

[0228] Optionally, for all or part of the pixels to be filtered in at least one image block, a region to be filtered is determined, which includes at least one of the pixels to be filtered, at least one neighboring pixel of the pixels to be filtered, and at least one non-neighboring pixel of the pixels to be filtered. The spatial features of the at least one image block are determined or obtained based on the statistical feature values ​​of the region to be filtered, and the at least one image block is filtered based on the spatial features of the at least one image block.

[0229] Optionally, a region to be filtered can be determined from an image block, which includes at least one of at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel. The spatial features of at least one image block can be determined or obtained based on the weighted statistical feature value of the region to be filtered, and the at least one image block can be filtered based on the spatial features of the at least one image block.

[0230] In this embodiment, by filtering at least one image block based on the spatial features of at least one image block, and comprehensively considering the spatial features determined or obtained based on the statistical feature values ​​of the region to be filtered, which includes at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel, the key image block features that need to be processed in the image block are determined. Thus, the key image block features can be focused on during the filtering process, which can improve the filtering effect of filtering at least one image block.

[0231] Third Embodiment

[0232] Based on any of the above embodiments, a third embodiment is proposed.

[0233] In this embodiment, the image processing method, step S1, performs filtering processing on at least one image block based on the spatial characteristics of at least one image block, including at least one of methods nine to eleven:

[0234] Method 9: Based on at least one spatial feature and at least one lookup table, determine or obtain at least one spatial attention weight, and perform filtering processing on the intermediate element block corresponding to at least one image block based on the at least one spatial attention weight;

[0235] Optionally, at least one spatial attention weight can be determined or obtained based on at least one spatial feature of at least one image block and at least one lookup table, and the intermediate element block corresponding to at least one image block can be filtered based on the at least one spatial attention weight.

[0236] Optionally, spatial attention weights can be spatial-level self-attention weights.

[0237] Optionally, the intermediate element block may contain at least one intermediate element, which may be an element determined based on the pixels to be filtered in the image block. The intermediate element may be a filtered pixel, which may be a pixel obtained by filtering the pixels to be filtered.

[0238] Optionally, at least one pixel to be filtered in at least one image block may be pre-filtered to obtain an intermediate element block containing at least one intermediate element.

[0239] Optionally, the intermediate elements can be obtained by preprocessing the pixels to be filtered.

[0240] Alternatively, lookup tables are a common method for accelerating computation in embedded systems. For a complex function or a series of calculations, if the output value is placed in a lookup table, then all that is needed afterward is to retrieve the value, without having to perform the computation process. Therefore, lookup tables are effective when the computation time is longer than the memory access time.

[0241] Alternatively, the lookup table can be used to approximate the computation of a neural network or neural network module, such as a 3x3 convolutional layer. The neural network or neural network module is pre-trained, and the lookup table is obtained based on the input-output mapping of the trained model.

[0242] Optionally, the lookup table may include at least one spatial feature, at least one spatial attention weight, and the correspondence between the two.

[0243] Optionally, the lookup table may also include at least one spatial feature, at least one intermediate value, and the correspondence between the two.

[0244] Optionally, the lookup table may also include at least one intermediate value, at least one spatial attention weight, and the correspondence between the two.

[0245] Optionally, at least one spatial attention weight can be determined or obtained in at least one lookup table based on at least one spatial feature of at least one image patch.

[0246] Optionally, after determining at least one spatial feature of at least one image block, the at least one spatial feature can be input into at least one lookup table for searching. Since the lookup table includes the correspondence between at least one spatial feature and at least one spatial attention weight, the spatial attention weight that corresponds to the input at least one spatial feature can be found in the at least one lookup table and output. Then, the intermediate element block corresponding to the at least one image block can be filtered according to the output at least one spatial attention weight.

[0247] Optionally, at least one spatial feature can be input into at least one lookup table to find at least one intermediate value, and at least one spatial attention weight can be determined or obtained based on the at least one intermediate value. For example, at least one intermediate value can be input into at least one neural network and / or at least one activation function to output at least one spatial attention weight.

[0248] Optionally, at least one intermediate value can be determined or obtained based on at least one spatial feature. For example, at least one spatial feature can be input into at least one neural network and / or at least one activation function to output at least one intermediate value, and the intermediate value can be input into at least one lookup table to obtain at least one spatial attention weight.

[0249] Optionally, the intermediate element block corresponding to the image block may contain image block features of at least one channel.

[0250] Optionally, based on the spatial features of an image patch and at least one lookup table, the spatial attention weights corresponding to the image patch features of at least one channel can be determined or obtained, and the image patch features of at least one channel can be filtered based on the spatial attention weights corresponding to the image patch features of one channel.

[0251] In this embodiment, by determining or obtaining at least one spatial attention weight based on at least one spatial domain feature of at least one image block and at least one lookup table, and performing filtering processing on the intermediate element block corresponding to at least one image block based on at least one spatial attention weight, the key image block features that need to be processed in the image block can be determined through spatial attention, thereby focusing on the key image block features during filtering processing, which can improve the filtering effect of filtering at least one image block.

[0252] Method 10: Based on at least one spatial feature and at least one neural network, determine or obtain at least one spatial attention weight, and perform filtering processing on the intermediate element block corresponding to at least one image block based on the at least one spatial attention weight;

[0253] Optionally, at least one spatial attention weight can be determined or obtained based on at least one spatial feature of at least one image block and at least one neural network, and the intermediate element block corresponding to at least one image block can be filtered based on the at least one spatial attention weight.

[0254] Alternatively, the concept of intermediate element blocks can be referenced in method nine above.

[0255] Optionally, spatial attention weights can be spatial-level self-attention weights.

[0256] Optionally, the neural network in this embodiment can be a neural network based on fully connected layers, a neural network based on convolutional layers, a neural network based on Transformer, or a neural network based on a mixture of convolutional and fully connected layers and Transformer, etc.

[0257] Optionally, at least one spatial feature can be input into at least one neural network, and at least one spatial attention weight can be output.

[0258] Optionally, at least one spatial feature can be input into at least one neural network, outputting at least one intermediate value, and at least one spatial attention weight can be determined or obtained based on the at least one intermediate value. For example, at least one intermediate value can be input into at least one lookup table or at least one activation function, outputting at least one spatial attention weight.

[0259] Optionally, at least one intermediate value can be determined or obtained based on at least one spatial feature. For example, at least one spatial feature can be input into at least one lookup table or at least one activation function to output at least one intermediate value, and the intermediate value can be input into at least one neural network to output at least one spatial attention weight.

[0260] Optionally, at least one spatial feature can be input into at least one lookup table for lookup, an intermediate value can be output, and the intermediate value can be input into at least one neural network to output at least one spatial attention weight.

[0261] Optionally, the lookup table may include at least one spatial feature, at least one intermediate value, and the correspondence between the two. Since the lookup table includes the correspondence between at least one spatial feature and at least one intermediate value, an intermediate value that corresponds to the input at least one spatial feature can be found in the at least one lookup table and output. The output at least one intermediate value is then input into at least one neural network to obtain at least one spatial attention weight. The intermediate element block corresponding to at least one image patch is then filtered according to the at least one spatial attention weight.

[0262] Optionally, at least one spatial feature can be input into at least one neural network, an intermediate value can be output, and the intermediate value can be input into at least one lookup table for lookup, and at least one spatial attention weight can be output.

[0263] Optionally, the lookup table may include at least one intermediate value, at least one spatial attention weight, and the correspondence between the two. Since the lookup table includes the correspondence between at least one intermediate value and at least one spatial attention weight, after inputting at least one spatial feature into at least one neural network and outputting an intermediate value, the at least one intermediate value can be input into at least one lookup table to find at least one spatial attention weight that corresponds to the input at least one intermediate value, and the intermediate element block corresponding to at least one image patch can be filtered according to the at least one spatial attention weight.

[0264] In this embodiment, by determining or obtaining at least one spatial attention weight based on at least one spatial domain feature of at least one image block and at least one neural network, and performing filtering processing on the intermediate element block corresponding to at least one image block based on at least one spatial attention weight, the key image block features that need to be processed in the image block can be determined through spatial attention, thereby focusing on the key image block features during filtering processing, which can improve the filtering effect of filtering at least one image block.

[0265] Method 11: Based on at least one spatial feature and at least one activation function, determine or obtain at least one spatial attention weight, and perform filtering processing on the intermediate element block corresponding to at least one image block based on the at least one spatial attention weight.

[0266] Optionally, at least one spatial attention weight can be determined or obtained based on at least one spatial feature and at least one activation function of at least one image block, and the intermediate element block corresponding to at least one image block can be filtered based on the at least one spatial attention weight.

[0267] Alternatively, the concept of intermediate element blocks can be referenced in method nine above.

[0268] Optionally, spatial attention weights can be spatial-level self-attention weights.

[0269] Optionally, the activation function, also known as the activation function, often exists between the input and output layers of a neural network, and its purpose is to add some non-linear factors to the neural network.

[0270] Optionally, at least one spatial feature can be input into at least one activation function to output at least one spatial attention weight.

[0271] Optionally, at least one spatial feature can be input into at least one activation function to output at least one intermediate value, and at least one spatial attention weight can be determined or obtained based on the at least one intermediate value. For example, at least one intermediate value can be input into at least one lookup table or at least one neural network to output at least one spatial attention weight.

[0272] Optionally, at least one intermediate value can be determined or obtained based on at least one spatial feature. For example, at least one spatial feature can be input into at least one lookup table or at least one neural network to output at least one intermediate value, and the intermediate value can be input into at least one activation function to output at least one spatial attention weight.

[0273] Optionally, at least one spatial attention weight can be determined or obtained based on at least one spatial feature, at least one activation function, and at least one lookup table.

[0274] Optionally, at least one spatial feature can be input into at least one lookup table for lookup, an intermediate value can be output, and the intermediate value can be input into at least one activation function to output at least one spatial attention weight.

[0275] Optionally, the lookup table may include at least one spatial feature, at least one intermediate value, and the correspondence between the two. Since the lookup table includes the correspondence between at least one spatial feature and at least one intermediate value, an intermediate value that corresponds to the input at least one spatial feature can be found in at least one lookup table and output. Then, the output at least one intermediate value is input into at least one activation function to obtain at least one spatial attention weight.

[0276] Optionally, at least one spatial feature can be input into at least one activation function, the intermediate value can be output, and the intermediate value can be input into at least one lookup table for lookup, and at least one spatial attention weight can be output.

[0277] Optionally, the lookup table may include at least one intermediate value, at least one spatial attention weight, and the correspondence between the two. Since the lookup table includes the correspondence between at least one intermediate value and at least one spatial attention weight, after inputting at least one spatial feature into at least one activation function and outputting an intermediate value, the at least one intermediate value can be input into at least one lookup table to find at least one spatial attention weight that corresponds to the input at least one intermediate value.

[0278] Optionally, at least one spatial attention weight can be determined or obtained based on at least one spatial feature, at least one activation function, and at least one neural network.

[0279] Optionally, at least one spatial feature can be input into at least one activation function, an intermediate value can be output, and then the intermediate value can be input into at least one neural network to obtain at least one spatial attention weight. Alternatively, at least one spatial feature can be input into at least one neural network, an intermediate value can be output, and then the intermediate value can be input into at least one activation function to obtain at least one spatial attention weight.

[0280] Optionally, at least one spatial attention weight can be determined or obtained based on at least one spatial feature, at least one activation function, at least one neural network, and at least one lookup table.

[0281] In this embodiment, by determining or obtaining at least one spatial attention weight based on at least one spatial domain feature and at least one activation function of at least one image block, and performing filtering processing on the intermediate element block corresponding to at least one image block based on at least one spatial attention weight, the key image block features that need to be processed in the image block can be determined through spatial attention, thereby focusing on the key image block features during filtering processing, which can improve the filtering effect of filtering at least one image block.

[0282] Fourth embodiment

[0283] Based on any of the above embodiments, a fourth embodiment is proposed.

[0284] In this embodiment, the image processing method further includes the following method twelve:

[0285] Method 12: Based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block, determine or obtain at least one intermediate element block.

[0286] Optionally, at least one intermediate element block can be determined or obtained based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block, and the intermediate element block corresponding to the at least one image block can be filtered based on the spatial characteristics of the at least one image block.

[0287] Optionally, Method Twelve can be combined with any one of Methods One to Eleven above. For example, Method Twelve can be combined with Method Nine above, and step S1 includes determining or obtaining at least one intermediate element block based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block; determining or obtaining at least one spatial attention weight based on at least one spatial feature of at least one image block and at least one lookup table; and performing filtering processing on the intermediate element block corresponding to at least one image block based on the at least one spatial attention weight.

[0288] Optionally, the concept of a neural network can be referred to in Method 10 of the above embodiments, and the concept of a lookup table can be referred to in Method 9 of the above embodiments.

[0289] Optionally, the lookup table structure may include a structure of at least one lookup table, such as a serial structure including at least one lookup table, a parallel structure including at least one lookup table, or a tree structure including at least one lookup table, etc., without limitation.

[0290] Optionally, the lookup table can be a feature value lookup table, similar to a neural network, where the feature value reflects the distribution characteristics of the original pixel to be filtered, such as edge / texture / semantic intensity. It can also be an attention weight lookup table or a style feature lookup table. The lookup table can be a filtering mode lookup table, a filtered pixel lookup table, or a lookup table that needs to be applied in the intermediate process of filtering, etc.

[0291] Optionally, at least one pixel to be filtered in at least one image block can be filtered based on a neural network, a lookup table structure, and at least one item in the lookup table, and at least one intermediate element block can be determined or obtained based on the filtering result.

[0292] In this embodiment, by determining or obtaining at least one intermediate element block based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block, and by filtering the intermediate element block corresponding to the at least one image block based on the spatial characteristics of the at least one image block, the key image block features that need to be processed in the image block can be determined through spatial attention, thereby focusing on the key image block features during the filtering process, which can improve the filtering effect of filtering the at least one image block.

[0293] Optionally, the lookup table structure includes at least one of the following methods thirteen through twenty-nine:

[0294] Method 13 requires at least one lookup table;

[0295] Optionally, the lookup table structure may include at least one lookup table, or it may include at least one lookup table and at least one neural network module.

[0296] Optionally, the index corresponding to at least one pixel to be filtered can be determined, and the index can be input into at least one lookup table in the lookup table structure for searching. Based on the search result, at least one intermediate element in the intermediate element block can be determined or obtained.

[0297] In this embodiment, when the lookup table structure is at least one lookup table, at least one lookup table can be used to filter at least one pixel to be filtered. In this way, the lookup table can be selected for filtering according to different scenarios to improve the filtering effect and thus support the improvement of video encoding and / or decoding efficiency.

[0298] Method fourteen, requiring at least one neural network module;

[0299] Optionally, the neural network module can be a type of neural network, such as a neural network containing only 3x3 convolutional layers, or a neural network module having at least one convolutional layer.

[0300] Optionally, the lookup table structure may include at least one neural network module, or it may include at least one lookup table and at least one neural network module.

[0301] Optionally, at least one pixel to be filtered can be input into the neural network module, and at least one intermediate element in the intermediate element block can be output.

[0302] Optionally, the index corresponding to at least one pixel to be filtered can be determined, and the index can be input into at least one lookup table in the lookup table structure for searching. The search result can be input into the neural network module for processing, and at least one intermediate element in at least one intermediate element block can be determined or obtained based on the output of the neural network module.

[0303] Optionally, at least one pixel to be filtered can be input into a neural network module, at least one index can be determined based on the output of the neural network module, and at least one index can be input into at least one lookup table for searching. Based on the search result, at least one intermediate element in at least one intermediate element block can be determined or obtained.

[0304] In this embodiment, when the lookup table structure is at least one neural network module, at least one neural network module can be used to filter at least one pixel to be filtered. In this way, the neural network module can be selected for filtering according to different scenarios to improve the filtering effect and thus support the improvement of video encoding and / or decoding efficiency.

[0305] Method 15: There are at least two search branches, and at least one search branch has the same input as the other search branch;

[0306] Optionally, the lookup table structure may include at least two lookup branches, and each lookup branch may include at least one lookup table. The at least one lookup table included in each of the at least two lookup branches may be the same or different, and there is no restriction on this.

[0307] Optionally, at least two search branches may have the same input, such as at least one of the following: input pixel value, input number of pixels.

[0308] Optionally, the inputs of at least one lookup table in at least one lookup table in another lookup branch can be the same, for example, both can come from the same intermediate element block of filtering (e.g., the block to be filtered, or a reference image block determined or obtained based on the block to be filtered).

[0309] For example, the input of at least one lookup table in at least one lookup branch is two pixels, and the input of at least one lookup table in another lookup branch can also be two pixels. The two input pixels can be the pixels to be filtered, or intermediate filtering elements determined or obtained based on the pixels to be filtered, etc.

[0310] In this embodiment, when the inputs of at least two lookup branches in the lookup table structure are the same, multiple filtering processes can be performed based on at least one pixel to be filtered combined with at least two lookup branches in the lookup table structure that have the same input, thereby improving the filtering effect.

[0311] Method 16: At least two search branches, where the input of at least one search branch is determined or obtained based on the output of the other search branch;

[0312] Optionally, there are at least two search branches in the lookup table structure, and the inputs and outputs of the two search branches can have a certain correlation.

[0313] Optionally, the input of at least one search branch can be determined or obtained based on the output of another search branch. For example, the output of at least one search branch can be used as the input of another search branch, or the output range of at least one search branch can be used as the input range of another search branch, and so on.

[0314] Optionally, the input of at least one lookup table in at least one search branch can be determined or obtained based on the output of at least one lookup table in another search branch. For example, the output of at least one lookup table in at least one search branch can be used as the input of at least one lookup table in another search branch, or the output range of at least one lookup table in at least one search branch can be used as the input range of at least one lookup table in another search branch, and so on.

[0315] Optionally, the input of at least one search branch can be determined or obtained based on at least one pixel to be filtered. For example, the at least one pixel to be filtered can be used as the input of at least one search branch for filtering, or the index corresponding to the at least one pixel to be filtered can be used as the input of at least one search branch for filtering, and then the output of the at least one search branch can be used as the input of another search branch for filtering, until at least one intermediate element in at least one intermediate element block is finally obtained.

[0316] In this embodiment, when the input of at least one lookup branch is determined or obtained based on the output of the other lookup branch in at least two lookup branches of the lookup table structure, at least one pixel to be filtered can be filtered sequentially based on these at least two lookup branches, thereby realizing multiple filtering processes and improving the filtering effect.

[0317] Method 17: At least two search branches, and at least two search branches are located in at least one channel corresponding to at least one pixel to be filtered or the first intermediate element of the filter;

[0318] Alternatively, the channels in this embodiment can be used for lookup tables.

[0319] Optionally, a channel is a component of the feature map of an image patch in the depth dimension, used to describe the feature representation of the number of features in a specific dimension. Each channel represents a certain feature (such as texture, edge, derived distribution, etc.) extracted from the image patch (such as a prediction patch or reconstructed patch). For example, one channel might detect horizontal edges, while another channel might detect vertical edges, etc.

[0320] Optionally, at least one pixel to be filtered may correspond to multiple channels.

[0321] Optionally, the first intermediate element of the filter can be data generated during the filtering process of the pixel to be filtered, such as feature values, filtering modes, or other filtering parameters.

[0322] Optionally, the first intermediate element of the filter can be one type of intermediate element of the filter.

[0323] Optionally, intermediate filtering elements refer to the data generated during the filtering process of the pixel to be filtered. For example, when filtering the pixel using three sequential lookup tables, the outputs of the first and second lookup tables are both intermediate filtering elements. Alternatively, after preprocessing the pixel to be filtered using DCT transform, the resulting new pixel to be filtered can also be called an intermediate filtering element.

[0324] Optionally, at least one first filter intermediate element corresponds to multiple channels.

[0325] Optionally, in the lookup table structure, there are at least two lookup branches, and the at least two lookup branches are located in the same channel.

[0326] Optionally, at least two lookup branches can be used as a filter-like device to filter at least one pixel to be filtered or a first intermediate element of the same channel.

[0327] Optionally, when processing the pixel features (such as pixel value, pixel position, etc.) of at least one pixel to be filtered or the first intermediate element of filtering in the same channel, at least two search branches can be processed in parallel or in sequence, without any restriction.

[0328] Optionally, filtering can be performed on at least one pixel to be filtered or the first intermediate element of filtering in the same channel based on at least one lookup table in each of at least two lookup branches.

[0329] In this embodiment, by using at least two lookup branches in the lookup table structure, and with at least two lookup branches located in at least one channel corresponding to at least one pixel to be filtered or the first intermediate element of filtering, the pixel features (such as pixel value or pixel position) of at least one pixel to be filtered or the first intermediate element of filtering in the same channel can be filtered using at least two lookup branches. This demonstrates the implementation of multiple filtering processes on the pixel features of the same channel, thereby improving the filtering effect.

[0330] Method 18: At least two search branches, where the search table with at least two search branches is searched in parallel and / or serial manner.

[0331] Optionally, the lookup table structure includes at least two lookup branches, and the lookup methods of the at least two lookup branches are parallel lookup and / or serial lookup.

[0332] Optionally, the input elements corresponding to the lookup tables in at least two lookup branches can be determined simultaneously based on at least one pixel to be filtered or the first intermediate element of filtering, and then input into their respective lookup tables for parallel searching to obtain at least one intermediate element of filtering. Then, the subsequent filtering processing method is used to perform filtering processing to obtain the filtered pixel.

[0333] Optionally, the input element corresponding to the lookup table in at least one lookup branch can be determined or obtained based on at least one pixel to be filtered or the first intermediate element of the filter, and then input into the lookup table in the lookup branch for searching. After the search is completed, the search operation of the lookup table in another lookup branch is performed, that is, a serial search is performed until at least one intermediate element in at least one intermediate element block is finally determined or obtained.

[0334] In this embodiment, by using parallel search and / or serial search in at least two search branches of the lookup table structure, different search methods can be selected according to different scenarios when using at least two search branches of the lookup table structure to filter at least one pixel to be filtered, thereby improving the filtering effect.

[0335] Method 19: At least two lookup tables, where the input of at least one lookup table is determined or obtained based on the type, size parameters, and / or input range of the other lookup table;

[0336] Optionally, there can be multiple types of lookup tables. Multiple lookup tables can be of the same type or different types, such as a filtered pixel lookup table, a homography matrix lookup table, an eigenvalue lookup table, etc.

[0337] Optionally, the size parameters of the lookup table may include the width, height, perimeter, and area of ​​the lookup table.

[0338] Optionally, the input range of the lookup table can be set in advance, and the input ranges of multiple lookup tables can be the same or different.

[0339] Optionally, the lookup table structure includes at least two lookup tables. The input of one lookup table can be determined based on the type of the other lookup table. For example, if two consecutive lookup tables are of type filtered pixel lookup tables, the input range of one filtered pixel lookup table is greater than or equal to the input range of the other filtered pixel lookup table.

[0340] Optionally, the lookup table structure may contain at least two lookup tables, and the input of one lookup table may be determined based on the size parameters of the other lookup table. For example, the input range of the lookup table with the larger size parameter may be greater than or equal to the input range of the lookup table with the smaller size parameter.

[0341] Optionally, the lookup table structure may contain at least two lookup tables, and the input of one lookup table may be determined based on the input range of the other lookup table, such as the inputs of the two lookup tables being identical.

[0342] In this embodiment, by using at least two lookup tables in the lookup table structure, where the input of at least one lookup table is determined or obtained based on the type, size parameters, and / or input range of the other lookup table, multiple filtering processes can be performed on at least one pixel to be filtered using at least two lookup tables, thereby improving the filtering effect.

[0343] Method 20: At least two lookup tables, located in at least one channel corresponding to at least one pixel to be filtered or the first intermediate element of the filter;

[0344] Optionally, at least one pixel to be filtered corresponds to multiple channels, and at least one first filtering intermediate element corresponds to multiple channels.

[0345] Optionally, the lookup table structure may include at least two lookup tables, which may be located in at least one channel of at least one pixel to be filtered, that is, at least one lookup table exists in each channel of at least one pixel to be filtered, or at least two lookup tables exist in the same channel of at least one pixel to be filtered.

[0346] Optionally, the at least two lookup tables may be located in at least one channel of at least one first filter intermediate element, that is, at least one lookup table exists in each channel of at least one first filter intermediate element, or at least two lookup tables may exist in the same channel of at least one first filter intermediate element.

[0347] In this embodiment, by using at least two lookup tables in the lookup table structure, with at least two lookup tables located in at least one channel corresponding to at least one pixel to be filtered or the first intermediate element of the first filter, multiple filtering processes can be performed on at least one pixel to be filtered or the first intermediate element of the first filter in the same channel using at least two lookup tables, thereby improving the filtering effect.

[0348] Method 21: At least two lookup tables, where the lookup method for at least two lookup tables within the same channel is parallel lookup and / or serial lookup;

[0349] Optionally, in the lookup table structure, the lookup method of at least two lookup tables is parallel lookup. When performing a parallel lookup on at least two lookup tables, the input of the two lookup tables comes from the same block (i.e., image block or intermediate element block). By performing a one-time scan on the pixel block to be filtered or the intermediate element block, the indexes of at least two lookup tables can be obtained simultaneously, and then the at least two lookup tables can be performed in parallel based on the indexes of at least two lookup tables.

[0350] Optionally, the intermediate element block can be a block containing filtered intermediate elements, which can be described as described above and will not be repeated here.

[0351] Optionally, in the lookup table structure, when at least two lookup tables are searched in parallel, the inputs and outputs of the two lookup tables are not dependent on each other. That is, the inputs of at least two lookup tables can come from different blocks or from the same block; at least two lookup tables can be searched simultaneously by multiple tasks / processes / hardware.

[0352] Optionally, in the lookup table structure, the lookup method for at least two lookup tables is serial lookup. When performing a serial lookup on at least two lookup tables, the input and output of the two lookup tables are dependent on each other. The operation of one lookup table must be performed first before the operation of the second lookup table is completed.

[0353] Optionally, the lookup table structure may include at least two lookup tables, and at least two lookup tables exist in the same channel corresponding to at least one pixel to be filtered, wherein the lookup method of the at least two lookup tables is parallel lookup and / or serial lookup.

[0354] Optionally, at least two lookup tables exist in the same channel corresponding to at least one first filter intermediate element, and the lookup methods of the at least two lookup tables are parallel lookup and / or serial lookup.

[0355] Optionally, when processing the pixel to be filtered according to the lookup table structure, at least two lookup tables in the same channel can be used to perform parallel and / or serial searches on the index corresponding to at least one pixel to be filtered or the first intermediate element of the filter until the filtered pixel is finally obtained.

[0356] In this embodiment, by using parallel search and / or serial search in at least two search tables within the same channel in the lookup table structure, different search methods can be selected according to different scenarios when at least two search branches of the lookup table structure are used to filter at least one pixel to be filtered in the same channel, thereby improving the filtering effect.

[0357] Method 22: At least two lookup tables, and the lookup tables in at least two channels are searched in parallel and / or serial ways.

[0358] Optionally, the lookup table structure includes at least two lookup tables, and at least two channels corresponding to at least one pixel to be filtered contain lookup tables, and the lookup methods of the lookup tables in these two channels are parallel lookup and / or serial lookup.

[0359] Optionally, at least two channels corresponding to at least one first filter intermediate element have lookup tables, and the lookup tables in these two channels are searched in parallel and / or serial.

[0360] Optionally, when processing the pixel to be filtered according to the lookup table structure, the lookup tables in at least two channels can be used to perform parallel searches of the index corresponding to the pixel to be filtered, or a serial search can be used to sequentially call the lookup tables in at least two channels until the filtered pixel is finally obtained.

[0361] Optionally, when processing the first filtered intermediate element according to the lookup table structure, the lookup tables in at least two channels can be used simultaneously to perform parallel searches on the index corresponding to the first filtered intermediate element, or a serial search can be used to sequentially call the lookup tables in at least two channels until the filtered pixel is finally obtained.

[0362] In this embodiment, by using parallel search and / or serial search in at least two search tables of the lookup table structure for at least two channels, different search methods can be selected according to different scenarios when using at least two search branches of the lookup table structure to filter at least one pixel to be filtered in different channels, thereby improving the filtering effect.

[0363] Method 23: At least one search branch and at least one neural network module, wherein the input of the at least one neural network module is the same as the input of the at least one search branch;

[0364] Optionally, the neural network module may include a neural network, such as at least one convolutional layer, a 3x3 convolutional layer, etc.

[0365] Optionally, the input to at least one lookup branch in the lookup table structure is the same as the input to at least one neural network module.

[0366] Optionally, at least one pixel to be filtered or a first intermediate element of the filter can be simultaneously input into at least one search branch and at least one neural network module for parallel processing. The output results of the parallel processing of at least one search branch and at least one neural network module can be fused or weighted before subsequent filtering processing is performed until the final filtered pixel is obtained.

[0367] In this embodiment, by including at least one lookup branch and at least one neural network module with the same input in the lookup table structure, it is possible to use at least one lookup branch of the lookup table and at least one neural network module to jointly perform filtering processing on at least one pixel to be filtered or the first intermediate element of filtering. The filtering process combines the advantages of both the lookup table in the lookup branch and the neural network in the neural network module to improve the filtering effect.

[0368] Method 24: at least one search branch and at least one neural network module, wherein the input of the at least one neural network module is determined or obtained based on the output of the at least one search branch;

[0369] Optionally, the input of at least one neural network module in the lookup table structure can be determined or obtained based on the output of at least one lookup branch, and the input of at least one neural network module can be determined or obtained based on the output of at least one lookup table.

[0370] Optionally, the index corresponding to at least one pixel to be filtered or the first intermediate element of the filter can be input into the lookup table in at least one lookup branch to perform a lookup, obtain the output of at least one lookup branch, and use the output of at least one lookup branch as the input of at least one neural network module (or transform the output of at least one lookup branch, such as through weighted processing, to obtain the input of at least one neural network module), and input it into at least one neural network module for processing until all filtering processes are completed and the filtered pixel is obtained.

[0371] In this embodiment, by including at least one lookup branch and at least one neural network module in the lookup table structure, and determining or obtaining the input of the at least one neural network module based on the output of the at least one lookup branch, it is possible to use at least one lookup branch of the lookup table and at least one neural network module to jointly perform filtering processing on at least one pixel to be filtered or the first intermediate element of filtering. The filtering effect is improved by combining the advantages of both the lookup table in the lookup branch and the neural network in the neural network module.

[0372] Method 25: at least one lookup table and at least one neural network module, wherein the input of the at least one neural network module is determined or obtained based on the type, size parameters and / or input range of the at least one lookup table;

[0373] Optionally, the type, size parameters, and / or input range of at least one lookup table can be determined in accordance with the above method ten.

[0374] Optionally, the lookup table structure may include at least one lookup table and at least one neural network module. At least one pixel to be filtered or a first intermediate element of filtering can be input into the at least one lookup table for searching. Based on the search result, the input of at least one neural network module is determined or obtained and input into the at least one neural network module for processing until all filtering processes are completed and the filtered pixel is obtained.

[0375] Optionally, the input to at least one neural network module can be determined based on the type, size parameter, and / or input range of at least one lookup table. For example, if two different types of lookup tables have different output ranges, their corresponding inputs to at least one neural network module will also be different; similarly, if two lookup tables with different size parameters have different output ranges, their corresponding inputs to at least one neural network module will also be different.

[0376] In this embodiment, by including at least one lookup branch and at least one neural network module in the lookup table structure, and the input of the at least one neural network module is determined or obtained according to the type, size parameters and / or input range of the at least one lookup table, it is possible to use at least one lookup branch of the lookup table and at least one neural network module to jointly perform filtering processing on at least one pixel to be filtered or the first intermediate element of filtering. The filtering effect is improved by combining the advantages of both the lookup table in the lookup branch and the neural network in the neural network module.

[0377] Method 26: At least two lookup tables, where the output ranges of the two lookup tables are the same and / or the output ranges of the lookup tables are different;

[0378] Optionally, at least two lookup tables in the lookup table structure have the same output range.

[0379] Optionally, at least two lookup tables in the lookup table structure have different output ranges.

[0380] Optionally, the lookup table structure may contain at least three lookup tables, with two lookup tables having the same output range and two lookup tables having different output ranges.

[0381] For example, the lookup table structure includes lookup table 1, lookup table 2 and lookup table 3. Lookup table 1 and lookup table 2 have the same output range, while lookup table 3 has a different output range from lookup table 1 and lookup table 2.

[0382] Optionally, at least two lookup tables with different output ranges can be used to filter at least one pixel to be filtered or the first intermediate element of the filter; or at least two lookup tables with the same output range can be used to filter at least one pixel to be filtered or the first intermediate element of the filter.

[0383] In this embodiment, by including at least two lookup tables in the lookup table structure, and the output ranges of the at least two lookup tables being the same and / or different, it is possible to achieve more flexible filtering processing based on the output ranges of different lookup tables when filtering at least one pixel to be filtered or the first intermediate element of filtering according to the lookup table structure, thereby improving the filtering effect.

[0384] Method 27: At least two lookup tables, where the input ranges of the two lookup tables are the same and / or the input ranges of the lookup tables are different;

[0385] Optionally, at least two lookup tables in the lookup table structure have the same input range.

[0386] Optionally, at least two lookup tables in the lookup table structure have different input ranges.

[0387] Optionally, the lookup table structure may contain at least three lookup tables, with two lookup tables having the same input range and two lookup tables having different input ranges.

[0388] For example, the lookup table structure includes lookup table 1, lookup table 2 and lookup table 3. Lookup table 1 and lookup table 2 have the same input range, while lookup table 3 has a different input range than both lookup table 1 and lookup table 2.

[0389] Optionally, at least two lookup tables with different input ranges can be used to filter at least one pixel to be filtered or the first intermediate element of the filter; or at least two lookup tables with the same input range can be used to filter at least one pixel to be filtered or the first intermediate element of the filter.

[0390] In this embodiment, by including at least two lookup tables in the lookup table structure, and having the same and / or different input ranges for the at least two lookup tables, it is possible to perform filtering more flexibly based on the different input ranges of the lookup tables when filtering at least one pixel to be filtered or the first intermediate element of the filter according to the lookup table structure, thereby improving the filtering effect.

[0391] Method 28: At least one first residual module based on a lookup table, the first residual module including a first branch containing at least one lookup table structure, an adder, and a second branch containing a short-circuit structure;

[0392] Optionally, a residual structure can be set in the lookup table structure, that is, at least one first residual module based on the lookup table can be set, and the residual structure of the lookup table can be reflected through the first residual module.

[0393] Optionally, a first residual module can be used to process at least one pixel to be filtered or a first intermediate element of filtering. For example, the first branch containing at least one lookup table structure and the second branch containing a short-circuit structure are used to process at least one pixel to be filtered or a first intermediate element of filtering respectively. Then, the output results of the first branch and the second branch are weighted and added together by an adder until all filtering processes are completed and the filtered pixel is obtained.

[0394] In this embodiment, by including at least one first residual module based on the lookup table structure, the first residual module includes a first branch containing at least one lookup table structure, an adder, and a second branch containing a short-circuit structure, it is possible to combine the residual structure with the filtering process when filtering at least one pixel to be filtered or the first intermediate element of the filter according to the lookup table structure, so as to reflect the advantages of the residual structure and improve the filtering effect of the filtering process.

[0395] Method 29: At least one second residual module based on a lookup table, the first input element of the second residual module passes through a first branch containing at least one lookup table structure to obtain a fourth output element, and the first input element of the third residual module passes through a second branch containing a short-circuit structure to obtain a fifth output element that is the same as the first input element. The fourth output element and the fifth output element are weighted and added by an adder to obtain a sixth output element.

[0396] Optionally, a residual structure can be set in the lookup table structure, such as at least one second residual module based on the lookup table, and / or at least one first residual module based on the lookup table.

[0397] Optionally, the first residual module can be the second residual module.

[0398] Optionally, the first input element (such as a pixel or index marker) of the second residual module can be determined or obtained based on at least one pixel to be filtered.

[0399] Optionally, in the first branch of the second residual module, the first input element can be input into at least one lookup table structure for processing, such as inputting its corresponding index into the lookup table for searching, and the search result is used as the fourth output element of the first branch.

[0400] Optionally, in the second branch of the second residual module, the first input element can be input into the second branch containing the short-circuit result, and the output can be the same as the first input element.

[0401] The fourth output element from the first branch and the fifth output element from the second branch are input into the adder for weighted summation to obtain the sixth output element.

[0402] Optionally, the sixth output element can be directly output as a filtered pixel, or it can be used as an intermediate filtering element to continue subsequent filtering processes until all filtering processes are completed and the filtered pixel is obtained.

[0403] In this embodiment, by including at least one second residual module based on the lookup table structure, the first input element of the second residual module passes through a first branch containing at least one lookup table structure to obtain a fourth output element, and the first input element of the third residual module passes through a second branch containing a short-circuit structure to obtain a fifth output element identical to the first input element. The fourth and fifth output elements are weighted and added by an adder to obtain a sixth output element. This allows for filtering of at least one pixel to be filtered or a first intermediate element based on the lookup table structure, while also incorporating the residual structure to demonstrate its advantages and improve the filtering effect.

[0404] Fifth embodiment

[0405] Based on any of the above embodiments, a fifth embodiment is proposed.

[0406] In this embodiment, the image processing method further includes at least one of the following methods 30 to 33:

[0407] Method 30: In at least one intermediate element block, at least two first intermediate elements have the same spatial attention weight and / or have different spatial attention weights.

[0408] Optionally, for at least two first intermediate elements in at least one intermediate element block, spatial attention weights corresponding to the at least two first intermediate elements can be determined or obtained, and the spatial attention weights corresponding to the at least two first intermediate elements can be different.

[0409] Optionally, method 30 can be combined with at least one of methods 1 to 29 in the above embodiments. For example, method 30 can be combined with method 9 above, and step S1 includes: for at least two first intermediate elements in at least one intermediate element block, spatial attention weights corresponding to at least two first intermediate elements can be determined or obtained based on at least one spatial feature and at least one lookup table. The spatial attention weights corresponding to at least two first intermediate elements can be different. The at least one intermediate element block is filtered based on the spatial attention weights corresponding to at least two first intermediate elements.

[0410] For example, the size parameter of the intermediate element block is 3×3, and the spatial attention weights corresponding to at least two of the nine first intermediate elements in the intermediate element block can be different. For example, the spatial attention weights corresponding to the nine first intermediate elements in the image block are all different.

[0411] Optionally, the spatial attention weights corresponding to the first intermediate element in at least two rows or at least two columns of the intermediate element block can be different. For example, the spatial attention weight corresponding to the first intermediate element in the first row can be different from the spatial attention weight corresponding to the first intermediate element in the second row, or the spatial attention weight corresponding to the first intermediate element in each row can be different.

[0412] Optionally, for at least two first intermediate elements in at least one intermediate element block, spatial attention weights corresponding to the at least two first intermediate elements can be determined or obtained, and the spatial attention weights corresponding to the at least two first intermediate elements can be the same.

[0413] For example, the size parameter of the intermediate element block is 3×3, and the spatial attention weights corresponding to at least two of the nine first intermediate elements in the intermediate element block can be the same. For example, the spatial attention weights corresponding to all nine first intermediate elements in the intermediate element block can be the same.

[0414] Optionally, the spatial attention weights corresponding to the first intermediate element in at least two rows or at least two columns of the intermediate element block can be the same. For example, the spatial attention weight corresponding to the first intermediate element in the first row can be the same as the spatial attention weight corresponding to the first intermediate element in the second row, or the spatial attention weight corresponding to the first intermediate element in each row can be the same.

[0415] Optionally, for at least three first intermediate elements in at least one intermediate element block, spatial attention weights corresponding to the at least three first intermediate elements can be determined or obtained. The spatial attention weights corresponding to at least two first intermediate elements can be different, and the spatial attention weights corresponding to at least two first intermediate elements can be the same.

[0416] For example, the size parameter of the intermediate element block is 3×3. Among the 9 first intermediate elements in the intermediate element block, at least two of the first intermediate elements can have the same spatial attention weight, and at least two of the first intermediate elements can have different spatial attention weights. For example, the spatial attention weights of the 4 first intermediate elements in the intermediate element block are all the same, and the spatial attention weights of the other 5 first intermediate elements are all different.

[0417] Optionally, the spatial attention weights corresponding to the first intermediate elements in at least two rows or at least two columns of the intermediate element block can be the same, or the spatial attention weights corresponding to the first intermediate elements in at least two rows or at least two columns can be different. For example, the spatial attention weights corresponding to the first intermediate elements in the first row and the first intermediate elements in the second row can be the same, or the spatial attention weights corresponding to the first intermediate elements in the second row and the first intermediate elements in the third row can be different.

[0418] Optionally, based on at least one spatial feature of at least one intermediate element block and at least one of at least one lookup table, at least one activation function and at least one neural network, the spatial attention weights corresponding to at least two first intermediate elements in the at least one intermediate element block can be determined or obtained. The spatial attention weights corresponding to the at least two first intermediate elements can be different. The at least one intermediate element block is filtered based on the different spatial attention weights corresponding to the at least two first intermediate elements.

[0419] In this embodiment, for at least two first intermediate elements in at least one intermediate element block, spatial attention weights corresponding to the at least two first intermediate elements can be determined or obtained. The spatial attention weights corresponding to the at least two first intermediate elements can be different and / or the same. Filtering the at least one intermediate element block based on the at least one spatial attention weight can determine the key image block features that need to be processed in the intermediate element block by using different and / or the same spatial attention weights. In this way, the key image block features can be focused during the filtering process, which can improve the filtering effect of filtering the at least one intermediate element block.

[0420] Method 31: In at least one intermediate element block, the spatial attention weight corresponding to at least one first intermediate element is greater than the spatial attention weight corresponding to the other first intermediate element.

[0421] Optionally, for at least two first intermediate elements in at least one intermediate element block, spatial attention weights corresponding to the at least two first intermediate elements can be determined or obtained, wherein the spatial attention weight corresponding to at least one first intermediate element is greater than the spatial attention weight corresponding to the other first intermediate element.

[0422] Optionally, method 31 can be combined with at least one of methods 1 to 29 in the above embodiments. For example, method 31 combined with method 9 above, step S1 includes, for at least two first intermediate elements in at least one intermediate element block, determining or obtaining the spatial attention weights corresponding to the at least two first intermediate elements based on at least one spatial feature and at least one lookup table, wherein the spatial attention weight corresponding to at least one first intermediate element is greater than the spatial attention weight corresponding to the other first intermediate element, and filtering the at least one intermediate element block based on the spatial attention weights corresponding to the at least two first intermediate elements.

[0423] For example, the size parameter of the intermediate element block is 3×3. The spatial attention weight corresponding to at least one of the nine first intermediate elements in the intermediate element block can be different from the spatial attention weight corresponding to the other first intermediate element. For example, the spatial attention weight corresponding to one first intermediate element in the intermediate element block is greater than the spatial attention weight corresponding to the other eight first intermediate elements.

[0424] Optionally, the spatial attention weight corresponding to the first intermediate element in at least one row or column of the intermediate element block may be greater than the spatial attention weight corresponding to the first intermediate element in another row or column. For example, the spatial attention weight corresponding to the first intermediate element in the first row may be greater than the spatial attention weight corresponding to the first intermediate element in the second row.

[0425] Optionally, based on at least one spatial feature and at least one of at least one lookup table, at least one activation function, and at least one neural network, spatial attention weights corresponding to at least two first intermediate elements in at least one intermediate element block can be determined or obtained, wherein the spatial attention weight corresponding to at least one first intermediate element is greater than the spatial attention weight corresponding to the other first intermediate element, and the at least one intermediate element block is filtered based on the spatial attention weights corresponding to at least two first intermediate elements.

[0426] In this embodiment, for at least two first intermediate elements in at least one intermediate element block, spatial attention weights corresponding to the at least two first intermediate elements can be determined or obtained. The spatial attention weight corresponding to at least one first intermediate element is greater than the spatial attention weight corresponding to the other first intermediate element. Filtering the at least one intermediate element block based on the at least one spatial attention weight can determine the key image block features that need to be processed in the intermediate element block by using spatial attention weights of different sizes. In this way, the key image block features can be focused during the filtering process, which can improve the filtering effect of filtering the at least one intermediate element block.

[0427] Method 32: The spatial attention weight corresponding to the intermediate element region containing at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to the intermediate element region containing at least one other first intermediate element.

[0428] Optionally, the intermediate element block includes at least one intermediate element region, and the intermediate element region includes at least one first intermediate element. Optionally, an intermediate element region may correspond to at least one spatial attention weight, or at least one intermediate element region may correspond to one spatial attention weight.

[0429] Optionally, for at least two intermediate element regions in at least one intermediate element block, spatial attention weights corresponding to the at least two intermediate element regions can be determined or obtained, wherein the spatial attention weight corresponding to the intermediate element region containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element region containing at least one other first intermediate element.

[0430] Optionally, method 32 can be combined with at least one of methods 1 to 29 in the above embodiments. For example, method 32 combined with method 9 above, step S1 includes, for at least two intermediate element regions in at least one intermediate element block, determining or obtaining the spatial attention weights corresponding to the at least two intermediate element regions based on at least one spatial feature and at least one lookup table, wherein the spatial attention weight corresponding to the intermediate element region containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element region containing at least one other first intermediate element, and filtering the at least one intermediate element block based on the spatial attention weights corresponding to the at least two intermediate element regions.

[0431] For example, the size parameter of the middle element block is 3×3. The middle element block contains two middle element regions. One middle element region contains the first middle element of the first row and the second row, and the other middle element region contains the first middle element of the third row. The spatial attention weight corresponding to the middle element region containing the first middle element of the first row and the second row can be greater than the spatial attention weight corresponding to the middle element region containing the first middle element of the third row.

[0432] Optionally, based on at least one spatial feature and at least one of at least one lookup table, at least one activation function, and at least one neural network, the spatial attention weights corresponding to at least two intermediate element regions in at least one intermediate element block can be determined or obtained. The spatial attention weight corresponding to the intermediate element region containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element region containing at least one other first intermediate element. The at least one intermediate element block is then filtered based on the spatial attention weights corresponding to the at least two intermediate element regions.

[0433] In this embodiment, for at least two intermediate element regions in at least one intermediate element block, spatial attention weights corresponding to the at least two intermediate element regions can be determined or obtained. The spatial attention weight corresponding to the intermediate element region containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element region containing at least one other first intermediate element. Filtering the at least one intermediate element block based on the at least one spatial attention weight can determine the key image block features that need to be processed in the intermediate element block by using spatial attention weights of different sizes. In this way, the key image block features can be focused during the filtering process, which can improve the filtering effect of filtering the at least one intermediate element block.

[0434] Method 33: The spatial attention weight corresponding to the intermediate element sub-block that contains at least one first intermediate element in at least one intermediate element block is greater than the spatial attention weight corresponding to the intermediate element sub-block that contains at least one other first intermediate element.

[0435] Optionally, the intermediate element block includes at least one intermediate element region, the intermediate element region includes at least one intermediate element sub-block, and the intermediate element sub-block includes at least one first intermediate element.

[0436] Optionally, an intermediate element sub-block may correspond to at least one spatial attention weight, or at least one intermediate element sub-block may correspond to one spatial attention weight.

[0437] Optionally, for at least two intermediate element sub-blocks in at least one intermediate element block, spatial attention weights corresponding to the at least two intermediate element sub-blocks can be determined or obtained, wherein the spatial attention weight corresponding to the intermediate element sub-block containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element sub-block containing at least one other first intermediate element.

[0438] Optionally, method 33 can be combined with at least one of methods 1 to 29 in the above embodiments. For example, method 33 combined with method 9 above, step S1 includes, for at least two intermediate element sub-blocks in at least one intermediate element block, determining or obtaining the spatial attention weights corresponding to the at least two intermediate element sub-blocks based on at least one spatial feature and at least one lookup table, wherein the spatial attention weight corresponding to the intermediate element sub-block containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element sub-block containing at least one other first intermediate element, and filtering the at least one intermediate element block based on the spatial attention weights corresponding to the at least two intermediate element sub-blocks.

[0439] For example, if the size parameter of the intermediate element block is 5×5, and the intermediate element block contains a 3×3 intermediate element region, the intermediate element region contains two intermediate element sub-blocks. One intermediate element sub-block contains the first intermediate element of the first row and the second row of the intermediate element region, and the other intermediate element sub-block contains the first intermediate element of the third row of the intermediate element region. The spatial attention weight corresponding to the intermediate element sub-block containing the first intermediate element of the first row and the second row can be greater than the spatial attention weight corresponding to the intermediate element sub-block containing the first intermediate element of the third row.

[0440] Optionally, based on at least one spatial feature of at least one intermediate element block and at least one of at least one lookup table, at least one activation function, and at least one neural network, the spatial attention weights corresponding to at least two intermediate element sub-blocks in the at least one intermediate element block can be determined or obtained. The spatial attention weight corresponding to the intermediate element sub-block containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element sub-block containing at least one other first intermediate element. Based on the spatial attention weights corresponding to at least two intermediate element sub-blocks, the at least one intermediate element block is filtered.

[0441] In this embodiment, for at least two intermediate element sub-blocks in at least one intermediate element block, spatial attention weights corresponding to the at least two intermediate element sub-blocks can be determined or obtained. The spatial attention weight corresponding to the intermediate element sub-block containing at least one first intermediate element is greater than the spatial attention weight corresponding to the intermediate element sub-block containing at least one other first intermediate element. Filtering the at least one intermediate element block based on the at least one spatial attention weight can determine the key image block features that need to be processed in the intermediate element block by using spatial attention weights of different sizes. In this way, the key image block features can be focused during the filtering process, which can improve the filtering effect of filtering the at least one intermediate element block.

[0442] Sixth Embodiment

[0443] Based on any of the above embodiments, a sixth embodiment is proposed.

[0444] In this embodiment, filtering is performed on the intermediate element block corresponding to at least one image block based on at least one spatial attention weight, including at least one of the following methods 34 to 36:

[0445] Method 34: Based on the spatial attention enhancement of the first intermediate element in at least one intermediate element block according to at least one spatial attention weight, determine or obtain at least one filter block or filter intermediate element;

[0446] Optionally, step S1 includes determining or obtaining at least one spatial attention weight based on at least one spatial feature of at least one image block, determining or obtaining at least one filter block or filter intermediate element based on the result of spatial attention enhancement of at least one first intermediate element based on at least one spatial attention weight, and determining or obtaining at least one filtered image block based on at least one filter block or filter intermediate element.

[0447] Optionally, method 34 can be combined with at least one of methods 1 to 33 in the above embodiments. For example, method 34 can be combined with method 30 in the above embodiments. Step S1 includes determining or obtaining spatial attention weights corresponding to at least two first intermediate elements based on at least one spatial feature of at least one image block. The spatial attention weights corresponding to the at least two first intermediate elements may be different. Based on the spatial attention weights corresponding to the at least two first intermediate elements of the at least one intermediate element block, the result of spatial attention enhancement is applied to the at least one first intermediate element to determine or obtain at least one filter block or filter intermediate element. Based on the at least one filter block or filter intermediate element, at least one filtered image block is determined or obtained.

[0448] Optionally, at least one spatial attention weight can be multiplied by at least one first intermediate element or an intermediate value obtained by processing at least one first intermediate element, and the spatial attention enhancement result can be determined or obtained based on the result of the multiplication.

[0449] Optionally, at least one filter block can be determined or obtained based on the result of spatial attention enhancement, and at least one filtered image block can be determined or obtained based on the at least one filter block. For example, at least one filtered image block contains at least one filter block.

[0450] Optionally, at least one intermediate filtering element can be determined or obtained based on the result of spatial attention enhancement, and the at least one intermediate filtering element can be processed, for example, based on at least one neural network, at least one activation function, and at least one item in at least one lookup table, and at least one filtered image patch can be determined or obtained based on the processing result. For example, at least one filtered image patch contains at least one processed intermediate filtering element.

[0451] Optionally, refer to Figure 7 Based on at least one image patch, at least one spatial feature can be determined or obtained. Based on at least one spatial feature, at least one spatial attention weight can be determined or obtained. Based on at least one image patch and at least one lookup table, at least one intermediate element block can be determined or obtained. Based on at least one spatial attention weight, the result of spatial attention enhancement on at least one intermediate element block can be used to determine or obtain at least one filter block or filter intermediate element. Based on at least one filter block or filter intermediate element, at least one filtered image patch can be determined or obtained.

[0452] In this embodiment, at least one spatial attention weight is determined or obtained based on at least one spatial domain feature of at least one image block. Based on the spatial attention enhancement result of at least one first intermediate element based on at least one spatial attention weight, at least one filter block or filter intermediate element is determined or obtained. Based on at least one filter block or filter intermediate element, at least one filtered image block is determined or obtained. The key image block features that need to be processed in the intermediate element block can be determined by the result of spatial attention enhancement of the first intermediate element based on the spatial attention weight. Thus, the key image block features can be focused during the filtering process, which can improve the filtering effect of filtering the at least one intermediate element block.

[0453] Method 35: Based on at least one spatial attention weight, determine or obtain at least one filter block or filter intermediate element by performing spatial attention enhancement on at least one intermediate element region in at least one intermediate element block.

[0454] Optionally, step S1 includes determining or obtaining at least one spatial attention weight based on at least one spatial feature of at least one image block, determining or obtaining at least one filter block or filter intermediate element based on the result of spatial attention enhancement of at least one intermediate element region in at least one intermediate element block based on at least one spatial attention weight, and determining or obtaining at least one filtered image block based on at least one filter block or filter intermediate element.

[0455] Optionally, method 35 can be combined with at least one of methods 1 to 33 in the above embodiments. For example, method 35 can be combined with method 30 in the above embodiments. Step S1 includes determining or obtaining spatial attention weights corresponding to at least two intermediate element regions based on at least one spatial feature of at least one image block. The spatial attention weights corresponding to the at least two intermediate element regions may be different. Based on the spatial attention weights corresponding to the at least two intermediate element regions of the at least one intermediate element block, the result of spatial attention enhancement is applied to at least one intermediate element region in the at least one intermediate element block to determine or obtain at least one filter block or filter intermediate element. Based on the at least one filter block or filter intermediate element, at least one filtered image block is determined or obtained.

[0456] Optionally, at least one spatial attention weight can be multiplied by at least one intermediate element region or an intermediate value obtained by processing at least one intermediate element region, and the spatial attention enhancement result can be determined or obtained based on the result of the multiplication.

[0457] Optionally, at least one filter block can be determined or obtained based on the result of spatial attention enhancement, and at least one filtered image block can be determined or obtained based on the at least one filter block. For example, at least one filtered image block contains at least one filter block.

[0458] Optionally, at least one intermediate filtering element can be determined or obtained based on the result of spatial attention enhancement, and the at least one intermediate filtering element can be processed, for example, based on at least one neural network, at least one activation function, and at least one item in at least one lookup table, and at least one filtered image patch can be determined or obtained based on the processing result. For example, at least one filtered image patch contains at least one processed intermediate filtering element.

[0459] In this embodiment, at least one spatial attention weight is determined or obtained based on at least one spatial feature of at least one image block. Based on the spatial attention enhancement result of at least one intermediate element region based on at least one spatial attention weight, at least one filter block or filter intermediate element is determined or obtained. Based on at least one filter block or filter intermediate element, at least one filtered image block is determined or obtained. The key image block features that need to be processed in the intermediate element block can be determined by the result of spatial attention enhancement of the intermediate element region based on the spatial attention weight. Thus, the key image block features can be focused during the filtering process, which can improve the filtering effect of filtering at least one intermediate element block.

[0460] Method 36: Based on at least one spatial attention weight, spatial attention enhancement is performed on at least one intermediate element region in at least one intermediate element block to determine or obtain at least one filter block or filter intermediate element.

[0461] Optionally, step S1 includes determining or obtaining at least one spatial attention weight based on at least one spatial feature of at least one image block, determining or obtaining at least one filter block or filter intermediate element based on the result of spatial attention enhancement of at least one intermediate element region in at least one intermediate element block based on at least one spatial attention weight, and determining or obtaining at least one filtered image block based on at least one filter block or filter intermediate element.

[0462] Optionally, method 36 can be combined with at least one of methods 1 to 33 in the above embodiments. For example, method 36 can be combined with method 30 in the above embodiments. Step S1 includes determining or obtaining spatial attention weights corresponding to at least two intermediate element regions based on at least one spatial feature of at least one image block. The spatial attention weights corresponding to the at least two intermediate element regions may be different. Based on the spatial attention weights corresponding to the at least two intermediate element regions of the at least one intermediate element block, the result of spatial attention enhancement is applied to at least one intermediate element region in the at least one intermediate element block to determine or obtain at least one filter block or filter intermediate element. Based on the at least one filter block or filter intermediate element, at least one filtered image block is determined or obtained.

[0463] Optionally, at least one spatial attention weight can be multiplied by at least one intermediate element region or an intermediate value obtained by processing at least one intermediate element region, and the spatial attention enhancement result can be determined or obtained based on the result of the multiplication.

[0464] Optionally, at least one filter block can be determined or obtained based on the result of spatial attention enhancement, and at least one filtered image block can be determined or obtained based on the at least one filter block. For example, at least one filtered image block contains at least one filter block.

[0465] Optionally, at least one intermediate filtering element can be determined or obtained based on the result of spatial attention enhancement, and the at least one intermediate filtering element can be processed, for example, based on at least one neural network, at least one activation function, and at least one item in at least one lookup table, and at least one filtered image patch can be determined or obtained based on the processing result. For example, at least one filtered image patch contains at least one processed intermediate filtering element.

[0466] In this embodiment, at least one spatial attention weight is determined or obtained based on at least one spatial feature of at least one image block. Based on the spatial attention enhancement result of at least one intermediate element region based on at least one spatial attention weight, at least one filter block or filter intermediate element is determined or obtained. Based on at least one filter block or filter intermediate element, at least one filtered image block is determined or obtained. The key image block features that need to be processed in the intermediate element block can be determined by the result of spatial attention enhancement of the intermediate element region based on the spatial attention weight. Thus, the key image block features can be focused during the filtering process, which can improve the filtering effect of filtering at least one intermediate element block.

[0467] Seventh Embodiment

[0468] Based on any of the above embodiments, a seventh embodiment is proposed.

[0469] In this embodiment, the image processing method further includes determining or obtaining the spatial features of at least one image block after image preprocessing.

[0470] Optionally, at least one spatial feature can be determined or obtained based on at least one image block after image preprocessing, and filtering processing can be performed on at least one image block or at least one image block after image preprocessing based on the at least one spatial feature.

[0471] Optionally, this embodiment can be combined with at least one of the methods one to thirty-six in the above embodiments.

[0472] In this embodiment, by determining or obtaining at least one spatial feature based on at least one image block after image preprocessing, and performing filtering processing on at least one image block or at least one image block after image preprocessing based on at least one spatial feature, it is possible to comprehensively consider the spatial features of at least one image block when performing filtering processing on at least one image block, so as to determine the key image block features that need to be processed in the image block, and thus focus on the key image block features during filtering processing, thereby improving the filtering effect of filtering at least one image block.

[0473] Optionally, image preprocessing includes at least one of the following methods 37 to 40:

[0474] Method 37: Image Sharpening;

[0475] Optionally, the spatial features of at least one image block after image sharpening can be determined or obtained, and the at least one image block or the at least one image block after image sharpening can be filtered based on the at least one spatial feature.

[0476] Optionally, method 37 can be combined with at least one of methods 1 to 36 in the above embodiments. For example, method 37 combined with method 1 in the above embodiments, step S1 includes determining or obtaining the spatial features of at least one image block after image sharpening, wherein the spatial features are determined or obtained based on the statistical feature values ​​of the pixel to be filtered and at least one neighboring pixel in the at least one image block after image sharpening, and filtering is performed on the at least one image block or the at least one image block after image sharpening based on the at least one spatial feature.

[0477] Optionally, for video compression tasks, since more high-frequency information such as texture is lost, a 3x3 Unsharp Mask (USM) sharpening technique can be applied to image patches to enhance texture intensity as subsequent prior input. The USM-sharpened image is denoted as... right Spatial features are extracted at each position (i,j), such as Figure 8 As shown.

[0478] Optionally, for position (i,j), extract the surrounding 5×5 window. To utilize the features of the entire window as much as possible, divide the window into inner and external regions, i.e., perform the Mean module operation in the graph. Then, average the features of these two regions to obtain the spatial features. Optionally, calculate the feature mean of position (i,j) and its above, below, left, and right positions, that is, calculate the feature mean of positions (i-1,j), (i+1,j), (i,j-1), and (i,j+1), denoted as a. innerCalculate the feature mean of the eight positions at the outermost layer of the window, that is, calculate the feature mean of positions (i-2,j-1), (i-2,j+1), (i-1,j-2), (i+1,j-2), (i+2,j-1), (i+2,j+1), (i-1,j+2), and (i+1,j+2), denoted as a. exter , (a inner a exter Let (i,j) be the spatial feature. Based on the spatial features of position (i,j), the spatial attention weights for position (i,j) can be further obtained:

[0479]

[0480] σ represents the sigmoid activation function; This can be either a Conv1x2 layer or a lookup table. Conv1x2 represents a 1x2 convolution kernel; style_feature is the spatial feature of the input, calculated by averaging the inner and external features. In other words, when obtaining the spatial attention weights at position (i,j) based on the spatial features, a convolutional layer (Conv1x2) or a lookup table is used, followed by a sigmoid layer. Since the output of the sigmoid function is between 0 and 1, it can be used as an attention weight / mask. The lookup table can be obtained by storing the input and output of the convolutional layer (Conv1x2).

[0481] Optionally, the result after spatial attention enhancement is:

[0482] f out =atten·f in +f in

[0483] f in It enhances the texture intensity as a priori input, and generates spatial attention weights through spatial feature extraction, further improving the network's ability to model spatial information and texture details.

[0484] In this embodiment, by determining or obtaining the spatial features of at least one image block after image sharpening, and performing filtering processing on at least one image block or at least one image block after image sharpening based on at least one spatial feature, excessive loss of texture details can be avoided through image sharpening. At the same time, when filtering at least one image block, the spatial features of at least one image block are comprehensively considered to determine the key image block features that need to be processed in the image block. Thus, the key image block features can be focused during the filtering process, which can improve the filtering effect of filtering at least one image block.

[0485] Method 38: Gradient Calculation;

[0486] Optionally, the spatial features of at least one image block after gradient calculation can be determined or obtained, and the at least one image block or the at least one image block after gradient calculation can be filtered based on the at least one spatial feature.

[0487] Optionally, method 38 can be combined with at least one of methods 1 to 36 in the above embodiments. For example, method 38 combined with method 1 in the above embodiments, step S1 includes determining or obtaining the spatial features of at least one image block after gradient calculation, wherein the spatial features are determined or obtained based on the statistical feature values ​​of the pixel to be filtered and at least one neighboring pixel in the at least one image block after gradient calculation, and filtering is performed on the at least one image block or the at least one image block after image sharpening based on the at least one spatial feature.

[0488] In this embodiment, the spatial features of at least one image block after gradient calculation are determined or obtained. Filtering is performed on at least one image block or at least one image block after gradient calculation based on the at least one spatial feature. This allows for comprehensive consideration of the spatial features of at least one image block during filtering to determine the key image block features that need to be processed. Consequently, the filtering process can focus on the key image block features, thereby improving the filtering effect of filtering at least one image block.

[0489] Method 39: Transformation;

[0490] Optionally, the spatial features of at least one transformed image block can be determined or obtained, and the at least one image block or the at least one transformed image block can be filtered based on the at least one spatial feature.

[0491] Alternatively, the transformation can be a Fourier transform, a wavelet transform, or a feature transformation network based on deep learning.

[0492] Alternatively, the transformation can also be a scale transformation, such as image scaling (bilinear interpolation, bicubic interpolation) or multi-scale pyramid construction (such as Gaussian pyramids, Laplacian pyramids).

[0493] Alternatively, the transformation can also be a linear transformation, a nonlinear transformation, or a projection transformation, such as the Hough transformation or perspective transformation.

[0494] Optionally, method 39 can be combined with at least one of methods 1 to 36 in the above embodiments. For example, method 39 combined with method 1 in the above embodiments, step S1 includes determining or obtaining the spatial domain features of at least one transformed image block, the spatial domain features being determined or obtained based on the statistical feature values ​​of the pixel to be filtered and at least one neighboring pixel in the at least one transformed image block, and performing filtering processing on at least one image block or at least one image block after image sharpening based on at least one spatial domain feature.

[0495] In this embodiment, by determining or obtaining the spatial features of at least one transformed image block, and performing filtering processing on at least one image block or at least one transformed image block based on the at least one spatial feature, it is possible to comprehensively consider the spatial features of at least one image block when performing filtering processing on at least one image block, so as to determine the key image block features that need to be processed in the image block, and thus focus on the key image block features during filtering processing, thereby improving the filtering effect of filtering processing on at least one image block.

[0496] Method 40 involves texture detection based on detection operators.

[0497] Optionally, the spatial features of at least one image block after texture detection based on the detection operator can be determined or obtained, and the at least one image block or the at least one image block after texture detection based on the at least one spatial feature can be filtered.

[0498] Optionally, the detection operator includes at least one of the Sobel operator, Scharr operator, Canny operator, and Laplacian operator.

[0499] Alternatively, the detection operator can also be a neural network-based operator.

[0500] Optionally, method forty can be combined with at least one of methods one to thirty-six in the above embodiments. For example, method forty combined with method one in the above embodiments, step S1 includes determining or obtaining the spatial features of at least one image block after texture detection based on the detection operator, wherein the spatial features are determined or obtained based on the statistical feature values ​​of the pixel to be filtered and at least one neighboring pixel in the at least one image block after texture detection based on the detection operator, and filtering is performed on the at least one image block or the at least one image block after image sharpening based on the at least one spatial feature.

[0501] In this embodiment, by determining or obtaining the spatial features of at least one image block after texture detection based on the detection operator, and performing filtering processing on at least one image block or at least one image block after texture detection based on the at least one spatial feature, it is possible to comprehensively consider the spatial features of at least one image block when performing filtering processing on at least one image block, so as to determine the key image block features that need to be processed in the image block, and thus focus on the key image block features during filtering processing, thereby improving the filtering effect of filtering processing on at least one image block.

[0502] Eighth embodiment

[0503] This application also provides a processing apparatus, referring to... Figure 9 The processing device includes:

[0504] Processing module A1 performs filtering processing on at least one image block based on the spatial characteristics of at least one image block.

[0505] Optionally, the spatial characteristics are determined or obtained based on at least one of the following:

[0506] Statistical feature values ​​of the pixel to be filtered and at least one neighboring pixel in at least one image block;

[0507] Statistical feature values ​​of at least two non-neighboring pixels in at least one image patch;

[0508] The statistical feature value of at least one of the following in at least one image block: the pixel to be filtered, the pixel above the pixel to be filtered, the pixel to the left of the pixel to be filtered, the pixel to the right of the pixel to be filtered, and the pixel below the pixel to be filtered;

[0509] The weighted statistical feature values ​​of the pixels to be filtered in at least one image block and the pixels adjacent to the pixels to be filtered;

[0510] The statistical feature value of at least one pixel in the outermost layer of the window centered on the pixel to be filtered, wherein the size parameter of the window is less than or equal to the size parameter of the image patch;

[0511] Statistical characteristic values ​​of eight pixels that are not adjacent to the pixel to be filtered;

[0512] At least one image block contains statistical feature values ​​of at least one of the following: at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel.

[0513] At least one image block contains statistical feature values ​​of the region to be filtered, including at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel.

[0514] Optionally, processing module A1 includes at least one of the following:

[0515] Based on at least one spatial feature and at least one lookup table, at least one spatial attention weight is determined or obtained, and the intermediate element block corresponding to at least one image block is filtered based on the at least one spatial attention weight.

[0516] Based on at least one spatial feature and at least one neural network, at least one spatial attention weight is determined or obtained, and the intermediate element block corresponding to at least one image block is filtered based on the at least one spatial attention weight.

[0517] Based on at least one spatial feature and at least one activation function, at least one spatial attention weight is determined or obtained, and the intermediate element block corresponding to at least one image block is filtered based on the at least one spatial attention weight.

[0518] Optionally, the processing module A1 further includes at least one of the following:

[0519] Based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block, determine or obtain at least one intermediate element block;

[0520] The lookup table structure includes at least one of the following:

[0521] At least one lookup table;

[0522] At least one neural network module;

[0523] There are at least two search branches, and at least one search branch has the same input as the other search branch;

[0524] There are at least two search branches, and the input of at least one search branch is determined or obtained based on the output of the other search branch;

[0525] At least two search branches, and at least two search branches are located in at least one channel corresponding to at least one pixel to be filtered or the first intermediate element of the filter;

[0526] There are at least two search branches, and the search method for the search table with at least two search branches is parallel search and / or serial search;

[0527] At least two lookup tables, and the input of at least one lookup table is determined or obtained based on the type, size parameters and / or input range of the other lookup table;

[0528] At least two lookup tables, and at least two lookup tables are located in at least one channel corresponding to at least one pixel to be filtered or the first intermediate element of the filter;

[0529] At least two lookup tables, and the lookup methods for at least two lookup tables within the same channel are parallel lookup and / or serial lookup;

[0530] At least two lookup tables, and the lookup tables in at least two channels are searched in parallel and / or serial manner;

[0531] At least one search branch and at least one neural network module, wherein the input of the at least one neural network module is the same as the input of the at least one search branch;

[0532] At least one search branch and at least one neural network module, wherein the input of the at least one neural network module is determined or obtained based on the output of the at least one search branch;

[0533] At least one lookup table and at least one neural network module, wherein the input of the at least one neural network module is determined or obtained based on the type, size parameters and / or input range of the at least one lookup table;

[0534] At least two lookup tables, at least two lookup tables have the same output range and / or the lookup tables have different output ranges;

[0535] At least two lookup tables, at least two lookup tables have the same input range and / or the lookup tables have different input ranges;

[0536] At least one first residual module based on a lookup table, the first residual module including a first branch containing at least one lookup table structure, an adder, and a second branch containing a short-circuit structure;

[0537] At least one second residual module based on a lookup table, the first input element of the second residual module is passed through a first branch containing at least one lookup table structure to obtain a fourth output element, and the first input element of the third residual module is passed through a second branch containing a short-circuit structure to obtain a fifth output element that is the same as the first input element. The fourth output element and the fifth output element are weighted and added by an adder to obtain a sixth output element.

[0538] Optionally, the processing module A1 further includes at least one of the following:

[0539] In at least one intermediate element block, at least two first intermediate elements have the same spatial attention weight and / or the first intermediate elements have different spatial attention weights.

[0540] In at least one intermediate element block, the spatial attention weight corresponding to at least one first intermediate element is greater than the spatial attention weight corresponding to the other first intermediate element.

[0541] The spatial attention weight corresponding to the region of at least one intermediate element that contains at least one first intermediate element is greater than the spatial attention weight corresponding to the region of at least one intermediate element that contains at least one other first intermediate element.

[0542] The spatial attention weight corresponding to the at least one intermediate element sub-block that contains at least one first intermediate element is greater than the spatial attention weight corresponding to the at least one intermediate element sub-block that contains at least one other first intermediate element.

[0543] Optionally, filtering is performed on the intermediate element block corresponding to at least one image patch based on at least one spatial attention weight, including at least one of the following:

[0544] Based on the spatial attention enhancement of the first intermediate element in at least one intermediate element block according to at least one spatial attention weight, at least one filter block or filter intermediate element is determined or obtained.

[0545] Based on at least one spatial attention weight, the spatial attention enhancement of at least one intermediate element region in at least one intermediate element block is performed to determine or obtain at least one filter block or filter intermediate element.

[0546] Based on at least one spatial attention weight, the spatial attention enhancement of at least one intermediate element sub-block in at least one intermediate element block is performed to determine or obtain at least one filter block or filter intermediate element.

[0547] Optionally, the processing module A1 further includes: determining or obtaining the spatial features of at least one image block after image preprocessing.

[0548] Optionally, image preprocessing includes at least one of the following:

[0549] Image sharpening;

[0550] Gradient calculation;

[0551] Transformation;

[0552] Texture detection is performed based on the detection operator.

[0553] The processing device provided in this application embodiment is similar in implementation principle and beneficial effect to the technical solution shown in the corresponding method embodiment above, and will not be described again here.

[0554] This application also provides a processing device, including a memory and a processor. The memory stores an image processing program, and when the image processing program is executed by the processor, it implements the steps of the image processing method in any of the above embodiments.

[0555] This application also provides a storage medium storing an image processing program, which, when executed by a processor, implements the steps of the image processing method in any of the above embodiments.

[0556] In the embodiments of the smart terminal and storage medium provided in this application, all the technical features of any of the above-described image processing method embodiments may be included. The extended and explanatory content of the specification is basically the same as that of the embodiments of the above methods, and will not be repeated here.

[0557] This application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to perform the methods described in the various possible implementations above.

[0558] This application also provides a chip, including a memory and a processor. The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that a device with the chip installed performs the methods described in the various possible implementations above.

[0559] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0560] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0561] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0562] The units in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0563] In this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions are generally described in detail only when they appear for the first time. When they appear again, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions that are not described in detail later can be referred to their previous relevant detailed descriptions.

[0564] In this application, the descriptions of the various embodiments have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0565] The technical features of the present application can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0566] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of this application.

[0567] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, storage disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0568] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image processing method, characterized in that, Including the following steps: S1, filter at least one image block based on the spatial characteristics of at least one image block; Step S1 includes at least one of the following: Based on at least one spatial feature and at least one lookup table, at least one spatial attention weight is determined or obtained, and the intermediate element block corresponding to at least one image block is filtered based on the at least one spatial attention weight. Based on at least one spatial feature and at least one neural network, at least one spatial attention weight is determined or obtained, and the intermediate element block corresponding to at least one image block is filtered based on the at least one spatial attention weight. Based on at least one spatial feature and at least one activation function, at least one spatial attention weight is determined or obtained, and the intermediate element block corresponding to at least one image block is filtered based on the at least one spatial attention weight. The step of filtering the intermediate element block corresponding to at least one image block based on at least one spatial attention weight includes: determining or obtaining at least one filter block based on the result of spatial attention enhancement of the first intermediate element in the at least one intermediate element block based on at least one spatial attention weight. Specifically, at least one image block is pre-filtered to obtain an intermediate element block; the spatial attention weight is a spatial self-attention weight; the first intermediate element in the intermediate element block is the filtering intermediate element in the intermediate element block, and the filtering intermediate element is the data generated during the filtering process of the pixel to be filtered.

2. The image processing method as described in claim 1, characterized in that, Spatial characteristics are determined or obtained based on at least one of the following: Statistical feature values ​​of the pixel to be filtered and at least one neighboring pixel in at least one image block; Statistical feature values ​​of at least two non-neighboring pixels in at least one image patch; The statistical feature value of at least one of the following in at least one image block: the pixel to be filtered, the pixel above the pixel to be filtered, the pixel to the left of the pixel to be filtered, the pixel to the right of the pixel to be filtered, and the pixel below the pixel to be filtered; The weighted statistical feature values ​​of the pixels to be filtered in at least one image block and the pixels adjacent to the pixels to be filtered; The statistical feature value of at least one pixel in the outermost layer of the window centered on the pixel to be filtered, wherein the size parameter of the window is less than or equal to the size parameter of the image patch; Statistical characteristic values ​​of eight pixels that are not adjacent to the pixel to be filtered; At least one image block contains statistical feature values ​​of at least one of the following: at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel. At least one image block contains statistical feature values ​​of the region to be filtered, including at least one pixel to be filtered, at least one neighboring pixel, and at least one non-neighboring pixel.

3. The image processing method as described in claim 1, characterized in that, It also includes at least one of the following: Based on at least one neural network, a lookup table structure, at least one item in the lookup table, and at least one image block, determine or obtain at least one intermediate element block; The lookup table structure includes at least one of the following: At least one lookup table; At least one neural network module; There are at least two search branches, and at least one search branch has the same input as the other search branch; There are at least two search branches, and the input of at least one search branch is determined or obtained based on the output of the other search branch; At least two search branches, and at least two search branches are located in at least one channel corresponding to at least one pixel to be filtered or the first intermediate element of the filter; There are at least two search branches, and the search method for the search table with at least two search branches is parallel search and / or serial search; At least two lookup tables, and the input of at least one lookup table is determined or obtained based on the type, size parameters and / or input range of the other lookup table; At least two lookup tables, and at least two lookup tables are located in at least one channel corresponding to at least one pixel to be filtered or the first intermediate element of the filter; At least two lookup tables, and the lookup methods for at least two lookup tables within the same channel are parallel lookup and / or serial lookup; At least two lookup tables, and the lookup tables in at least two channels are searched in parallel and / or serial manner; At least one search branch and at least one neural network module, wherein the input of the at least one neural network module is the same as the input of the at least one search branch; At least one search branch and at least one neural network module, wherein the input of the at least one neural network module is determined or obtained based on the output of the at least one search branch; At least one lookup table and at least one neural network module, wherein the input of the at least one neural network module is determined or obtained based on the type, size parameters and / or input range of the at least one lookup table; At least two lookup tables, at least two lookup tables have the same output range and / or the lookup tables have different output ranges; At least two lookup tables, at least two lookup tables have the same input range and / or the lookup tables have different input ranges; At least one first residual module based on a lookup table, the first residual module including a first branch containing at least one lookup table structure, an adder, and a second branch containing a short-circuit structure; At least one second residual module based on a lookup table, the first input element of the second residual module is passed through a first branch containing at least one lookup table structure to obtain a fourth output element, and the first input element of the third residual module is passed through a second branch containing a short-circuit structure to obtain a fifth output element that is the same as the first input element. The fourth output element and the fifth output element are weighted and added by an adder to obtain a sixth output element.

4. The image processing method as described in claim 1, characterized in that, It also includes at least one of the following: In at least one intermediate element block, at least two first intermediate elements have the same spatial attention weight and / or the first intermediate elements have different spatial attention weights. In at least one intermediate element block, the spatial attention weight corresponding to at least one first intermediate element is greater than the spatial attention weight corresponding to the other first intermediate element. The spatial attention weight corresponding to the region of at least one intermediate element that contains at least one first intermediate element is greater than the spatial attention weight corresponding to the region of at least one intermediate element that contains at least one other first intermediate element. The spatial attention weight corresponding to the at least one intermediate element sub-block that contains at least one first intermediate element is greater than the spatial attention weight corresponding to the at least one intermediate element sub-block that contains at least one other first intermediate element.

5. The image processing method as described in claim 1, characterized in that, Filtering the intermediate element block corresponding to at least one image block based on at least one spatial attention weight, further comprising at least one of the following: Based on at least one spatial attention weight, the spatial attention enhancement of at least one intermediate element region in at least one intermediate element block is performed to determine or obtain at least one filter block or filter intermediate element. Based on at least one spatial attention weight, the spatial attention enhancement of at least one intermediate element sub-block in at least one intermediate element block is performed to determine or obtain at least one filter block or filter intermediate element.

6. The image processing method according to any one of claims 1 to 5, characterized in that, Also includes: Determine or obtain the spatial features of at least one image patch after image preprocessing.

7. The image processing method as described in claim 6, characterized in that, Image preprocessing includes at least one of the following: Image sharpening; Gradient calculation; Transformation; Texture detection is performed based on the detection operator.

8. A processing device, characterized in that, include: A memory and a processor, wherein the memory stores an image processing program, and the image processing program, when executed by the processor, implements the steps of the image processing method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the image processing method as described in any one of claims 1 to 7.

Images