Image processing method, processing device and storage medium

Abstract

Provided in the present disclosure are an image processing method, a processing device and a storage medium. The image processing method comprises: using a neural network including at least one partial convolution module to perform filtering processing on at least one image block, so as to obtain a target image block (S10). By means of the technical solution in the present disclosure, when a neural network is used for filtering processing, the complexity of video coding and / or decoding can be reduced, thereby improving the efficiency of video coding and / or decoding.

Description

Image processing method, processing device and storage medium TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image processing method, a processing device and a storage medium. BACKGROUND

[0002] The existing high-efficiency video coding standard protocol (H.266 / VVC) proposes a video frame coding technology to improve the coding performance without significantly increasing the computational complexity.

[0003] In the process of conceiving and implementing the present application, the inventors have found that at least the following problems exist: In the loop filtering stage of the coding process, the LC-NNLF (Low Complexity Neural Network based Loop Filter) structure is introduced in the NNVC (Neural Network based Video Coding) to perform loop filtering processing. However, due to the CP decomposition (Candecomp Parafac) step in the LC-NNLF structure, the neural network training process is unstable, and the model does not converge, and / or the LC-NNLF ignores the channel redundancy of the convolution layer in the neural network, affecting the image quality enhancement effect in the filtering process, resulting in high complexity of video coding and / or decoding, and thus limiting the efficiency of video coding and / or decoding.

[0004] The foregoing description is directed to providing general background information and does not necessarily constitute the prior art.

[0005] SUMMARY

[0006] To solve the above technical problems, the present application provides an image processing method, a processing device and a storage medium, which aims to solve the technical problem of how to reduce the complexity of video coding and / or decoding when using a neural network for filtering processing, and thus improve the efficiency of video coding and / or decoding.

[0007] The present application provides an image processing method, which can be applied to a processing device, comprising the steps of:

[0008] S10, using a neural network comprising at least one partial convolution module to perform filtering processing on at least one image block to obtain a target image block.

[0009] Optionally, step S10 comprises the steps of:

[0010] S11, input at least one of the image blocks, the quantization parameter and the boundary strength into a neural network comprising at least one partial convolution module;

[0011] S12, perform convolution processing on the at least one image block according to a first convolution layer of the neural network to determine or obtain image block features to be input into the partial convolution module;

[0012] S13, perform convolution processing on the at least one image block feature according to the at least one partial convolution module to determine or obtain target fusion features;

[0013] S14, determine or obtain at least one target image block after filtering processing according to the target fusion features.

[0014] Optionally, step S13 comprises:

[0015] input the at least one image block feature into the at least one partial convolution module;

[0016] divide the at least one image block feature by channel or region to determine or obtain at least two first sub-feature sets of the image block;

[0017] perform convolution processing on the at least one first sub-feature set to determine or obtain a second sub-feature set;

[0018] determine or obtain the target fusion features according to the second sub-feature set and the first sub-feature set which has not been processed by convolution.

[0019] Optionally, dividing the at least one image block feature by channel or region comprises at least one of:

[0020] divide the at least one image block feature by channel or region according to parameters of the partial convolution module;

[0021] divide the at least one image block feature by channel according to a number of channels corresponding to at least one convolution layer in at least one hidden layer of the neural network;

[0022] determine or generate an image block segmentation mark according to the image block feature, and divide the at least one image block feature by region according to the image block segmentation mark.

[0023] Optionally, the image block feature comprises a chroma component image block feature and a luminance component image block feature.

[0024] Optionally, step S13 comprises:

[0025] perform convolution processing on the chroma component image block feature according to a partial convolution module corresponding to the chroma component to determine or obtain fusion features of the chroma component;

[0026] The luminance component image block features are convoluted by the partial convolution module corresponding to the luminance component to determine or obtain the fusion features of the luminance component.

[0027] The target fusion features are determined or obtained according to the fusion features of the luminance component and the fusion features of the chroma component.

[0028] Optionally, step S14 comprises:

[0029] The target fusion features are convoluted by the last convolution layer of the neural network;

[0030] The at least one target image block after filtering is determined or obtained according to the target fusion features after convolution and the at least one image block input to the neural network.

[0031] Optionally, the image processing method further comprises at least one of the following:

[0032] The image block features are convoluted by the partial convolution module on partial channels;

[0033] The image block features are convoluted by the partial convolution module on partial regions of the image block features;

[0034] The image block is a reconstruction of a block;

[0035] The partial convolution module comprises at least one branch with an identity mapping function;

[0036] The partial convolution module comprises at least one convolution layer branch.

[0037] Optionally, the image processing method further comprises at least one of the following:

[0038] The number of partial convolution modules in the neural network is determined according to the flag;

[0039] The number of branches in the partial convolution modules in the neural network is determined according to the flag;

[0040] The plurality of convolution layers in the partial convolution modules in the neural network in the training stage are fused to determine or obtain the partial convolution modules in the inference stage.

[0041] The application also provides a processing device comprising:

[0042] The processing module is configured to filter at least one image block by using a neural network comprising at least one partial convolution module to obtain target image blocks.

[0043] The application further provides a processing device, comprising a memory and a processor, wherein the memory stores an image processing program, and the processor executes the image processing program to implement the steps of the image processing method according to any one of the above.

[0044] The application further provides a storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the image processing method according to any one of the above.

[0045] The image processing method of the application can be applied to a processing device, which comprises filtering at least one image block by using a neural network comprising at least one partial convolution module to obtain a target image block. According to the technical solution of the application, when filtering at least one image block by using a neural network, the partial convolution module can be controlled to implement partial convolution, thereby reducing the complexity of the filter. And because the partial convolution module is controllable, the redundancy of the convolution features of the neural network can be fully considered, so that when filtering by using a neural network, the redundancy of the neural network can be reduced, thereby reducing the complexity of video encoding and / or decoding, and further improving the efficiency of video encoding and / or decoding. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application. In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, those drawings can also provide other drawings for those skilled in the art without any creative effort.

[0047] FIG. 1 is a schematic diagram of the hardware structure of a mobile terminal for implementing various embodiments of the application;

[0048] FIG. 2 is a schematic diagram of a communication network system architecture provided by the embodiments of the application;

[0049] FIG. 3 is a schematic diagram of the flow of the image processing method according to the first embodiment;

[0050] FIG. 4 is a schematic diagram of the architecture of the neural network using the CP decomposition strategy in the image processing method according to the first embodiment;

[0051] FIG. 5 is a schematic diagram of the architecture of the neural network comprising at least one partial convolution module in the image processing method according to the first embodiment;

[0052] FIG. 6 is a schematic diagram of the flow of encoding and decoding in the image processing method according to the first embodiment;

[0053] Fig. 7 is a flow diagram illustrating an image processing method according to a second embodiment;

[0054] Fig. 8 is a schematic diagram of a processing module of a processing device.

[0055] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments, with reference to the accompanying drawings. The above-described drawings have shown the specific embodiments of the present application, and will be described in more detail hereinafter. These drawings and the written description are not intended to restrict the scope of the present application in any way, but to explain the present application to those skilled in the art by reference to a particular embodiment. DETAILED DESCRIPTION

[0056] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. The following description is presented with reference to the accompanying drawings, in which a similar reference can indicate similar elements or features. The embodiments described in the following exemplary embodiments are not meant to be all inclusive of all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application, as detailed in the appended claims.

[0057] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element. In addition, components, features, elements with identical names in different embodiments of the present application can have the same meaning or different meanings, and the specific meaning thereof should be determined in the light of its explanation in the specific embodiment or further in the light of the context in the specific embodiment.

[0058] It should be understood that, although terms, first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a particular order or hierarchy among the information. These terms are used only to distinguish one category of information from another category of information. For example, a first information can be termed a second information, and similarly, a second information can be termed a first information without departing from the scope hereof. As used herein, the term, if, can be interpreted to mean, at the time of, when, or in response to a determination, depending on the context. Also, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It will be further understood that the terms "comprises", "comprising", "includes" and / or "including", when used herein, specify the presence of stated features, steps, operations, elements, components, items, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, items, and / or groups thereof. As used herein, the term "or", "and / or", "at least one of", and the like are to be interpreted as inclusive or meaning any one or any combination. For example, "A, B or C" or "A, B, and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Only when a group of elements, functions, steps or operations are in some way inherently mutually exclusive is an exception to this definition.

[0059] It should be understood that, although various steps in the flowcharts of the embodiments herein can be illustrated as sequential process, these steps need not be performed in the order illustrated. Unless explicitly stated, the order of execution of the steps can be changed, and the steps can be performed in other orders. Also, at least some of the steps can include multiple sub-steps or multiple stages, which need not be performed in a same time instant, but can be performed in different time instants, and need not be performed sequentially, but can be performed in parallel or in alternate with at least some of the steps or sub-steps or stages of other steps.

[0060] As used herein, the term "if' can be construed to mean "when" or "in response to a determination" or "in response to a detection," depending on the context. Similarly, the phrase "if determined" or "if detecting (a stated condition or event)" can be construed to mean "when determined" or "in response to a determination" or "when detecting (a stated condition or event)" or "in response to a detection (a stated condition or event)," depending on the context.

[0061] It should be noted that, in the present document, step codes such as S10, S20, etc. are adopted for the purpose of more clearly and briefly expressing the corresponding content, and do not constitute substantial limitations on the order, and a person skilled in the art may perform S20 before S10, etc. in the specific implementation, but these should be within the protection scope of the present application.

[0062] It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0063] In the following description, the suffixes such as "module", "part", or "unit" used for an element are merely intended for facilitating explanation of the present application, and are not intended to have the specific meaning. Thus, "module", "part", or "unit" can be mixedly used.

[0064] The processing device in the present application can be a smart terminal or a server. Alternatively, the smart terminal can be implemented in various forms. For example, the smart terminal described in the present application can include a smart terminal such as a mobile phone, a tablet computer, a notebook computer, a palmtop computer, a Personal Digital Assistant (PDA), a Portable Media Player (PMP), a navigation device, a wearable device, a smart bracelet, a pedometer, and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like.

[0065] In the following description, a mobile terminal will be exemplified, and a person skilled in the art will understand that the configuration according to the embodiments of the present application can be applied to a fixed type terminal, except for elements particularly used for mobile purposes.

[0066] Referring to FIG. 1, it is a schematic diagram of a hardware structure of a mobile terminal implementing various embodiments of the present application. The mobile terminal 100 can include an RF (Radio Frequency) unit 101, a WiFi module 102, an audio output unit 103, an A / V (audio / video) input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, a processor 110, and a power supply 111, etc. A person skilled in the art can understand that the mobile terminal structure shown in FIG. 1 does not constitute a limitation on the mobile terminal, and the mobile terminal can include more or less components than the diagram, or combine certain components, or different component arrangement.

[0067] The various components of the mobile terminal will be specifically introduced below in conjunction with FIG. 1:

[0068] The radio frequency unit 101 can be used for receiving and transmitting signals in the process of transmitting or receiving information or a call. Specifically, the radio frequency unit 101 receives downlink information from a base station and provides the received information to the processor 110 for processing. In addition, the radio frequency unit 101 transmits uplink data to the base station. Generally, the radio frequency unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, and the like. In addition, the radio frequency unit 101 can communicate with a network and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System for Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution), TDD-LTE (Time Division Duplexing-Long Term Evolution), 5G, and 6G, and the like.

[0069] WiFi belongs to a short-range wireless transmission technology. The WiFi module 102 can help a user to send and receive e-mails, browse web pages, and access streaming media, and the like. The WiFi module 102 provides the user with wireless broadband Internet access. Although the WiFi module 102 is shown in FIG. 1, it can be understood that it does not belong to the essential components of the mobile terminal, and can be omitted as needed without changing the essence of the application.

[0070] The audio output unit 103 can convert audio data, which is received by the radio frequency unit 101 or the WiFi module 102 or stored in the memory 109, into an audio signal and output the audio signal as sound when the mobile terminal 100 is in a call signal reception mode, a call mode, a recording mode, a voice recognition mode, a broadcast reception mode, and the like. In addition, the audio output unit 103 can provide audio output related to a particular function performed by the mobile terminal 100 (e.g., a call signal reception sound, a message reception sound, and the like). The audio output unit 103 can include a speaker, a buzzer, and the like.

[0071] The A / V input unit 104 is configured to receive audio or video signals. The A / V input unit 104 can include a graphics processor (GPU) 1041 and a microphone 1042. The graphics processor 1041 processes image data of a still picture or a video obtained by an image capture device (e.g., a camera) in a video capture mode or an image capture mode. Processed image frames can be displayed on the display unit 106. Processed image frames can be stored in the memory 109 (or other storage medium) or transmitted via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) via the microphone 1042 in a telephone call mode, a recording mode, a voice recognition mode, or the like, and can process such sound into audio data. Processed audio (voice) data can be converted into a format that can be transmitted to a mobile communication base station via the radio frequency unit 101 in the case of the telephone call mode. The microphone 1042 can implement various types of noise cancellation (or suppression) algorithms to cancel (or suppress) noise or interference generated in the process of receiving and transmitting audio signals.

[0072] The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. The light sensor includes an ambient light sensor and a proximity sensor, which can optionally adjust the brightness of the display panel 1061 according to the brightness of ambient light, and turn off the display panel 1061 and / or the backlight when the mobile terminal 100 is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally, three axes), and detect the magnitude and direction of gravity when at rest, which can be used for applications that recognize the posture of the mobile terminal (such as switching between landscape and portrait screens, related games, magnetometer posture calibration), vibration recognition related functions (such as a pedometer, a knock), and the like. The mobile terminal can also be configured with a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and other sensors, which are not described herein.

[0073] The display unit 106 is configured to display information input by a user or information provided to the user. The display unit 106 can include a display panel 1061, which can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0074] The user input unit 107 can be used to receive input numerals or character information, and to generate key signal inputs related to user settings of the mobile terminal and control of functions. Optionally, the user input unit 107 can include a touch panel 1071 and other input devices 1072. The touch panel 1071, also called a touch screen, can collect touch operations of a user on or proximity thereto (such as operations of the user using a finger, a stylus, or any suitable object or accessory on or in proximity to the touch panel 1071), and drive corresponding connection devices according to a pre-set program. The touch panel 1071 can include two parts, a touch detecting device and a touch controller. The touch detecting device detects the user's touch position and detects signals resulting from touch operations, and transmits the signals to the touch controller; the touch controller receives touch information from the touch detecting device, and converts it into touch coordinates, and sends it to the processor 110, and can receive commands from the processor 110 and execute them. In addition, the touch panel 1071 can be implemented in various types such as a resistive type, a capacitive type, an infrared type, and a surface acoustic wave type. In addition to the touch panel 1071, the user input unit 107 can include other input devices 1072. Optionally, the other input devices 1072 can include one or more of, but are not limited to, a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.), a trackball, a mouse, a joystick, etc.

[0075] Optionally, the touch panel 1071 can cover the display panel 1061, and when the touch panel 1071 detects a touch operation on or in proximity thereto, it transmits to the processor 110 to determine the type of touch event, and then the processor 110 provides corresponding visual output on the display panel 1061 according to the type of touch event. Although in FIG. 1, the touch panel 1071 and the display panel 1061 are implemented as two independent components to realize the input and output functions of the mobile terminal, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize the input and output functions of the mobile terminal, without being limited specifically herein.

[0076] The interface unit 108 serves as an interface through which at least one external device can be connected with the mobile terminal 100. For example, the external device can include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, an audio input / output (I / O) port, a video I / O port, an earphone port, and the like. The interface unit 108 can be used to receive input (e.g., data information, power, and the like) from an external device and to transmit the received input to one or more elements within the mobile terminal 100, or can be used to transmit data between the mobile terminal 100 and the external device.

[0077] The memory 109 can be used to store software programs and various data. The memory 109 can mainly include a program storage area and a data storage area, and the program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), and the like. In addition, the memory 109 can include a high-speed random access memory, and can also include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0078] The processor 110 is a control center of the mobile terminal, connects all parts of the mobile terminal through various interfaces and lines, executes various functions of the mobile terminal and processes data by running or executing software programs and / or modules stored in the memory 109 and calling data stored in the memory 109, and thus performs overall monitoring on the mobile terminal. The processor 110 can include one or more processing units; preferably, the processor 110 can integrate an application processor and a modem processor, and the application processor can mainly process an operating system, a user interface, and application programs, and the like, and the modem processor can mainly process wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 110.

[0079] The mobile terminal 100 can also include a power supply 111 (such as a battery) for supplying power to various components; preferably, the power supply 111 can be logically connected to the processor 110 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system.

[0080] Although not shown in FIG. 1, the mobile terminal 100 can also include a Bluetooth module, and the like, which will not be described here.

[0081] In order to facilitate understanding of the embodiments of the present application, the communication network system based on the mobile terminal of the present application is described below.

[0082] Referring to FIG. 2, FIG. 2 is a communication network system architecture diagram provided by an embodiment of the present application, and the communication network system is an LTE system of a general mobile communication technology, which includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203, and an operator's IP service 204 which are sequentially connected in communication.

[0083] Optionally, the UE 201 can be the terminal 100 described above, which will not be repeated here.

[0084] The E-UTRAN 202 includes eNode Bs 2021 and other eNode Bs 2022. Optionally, the eNode Bs 2021 can be connected to each other through backhaul (for example, X2 interface), the eNode B 2021 is connected to the EPC 203, and the eNode B 2021 can provide access for the UE 201 to the EPC 203.

[0085] The EPC 203 can include MME (Mobility Management Entity) 2031, HSS (Home Subscriber Server) 2032, other MMEs 2033, SGW (Serving Gate Way) 2034, PGW (PDN Gate Way) 2035, and PCRF (Policy and Charging Rules Function) 2036. Optionally, the MME 2031 is a control node for processing signaling between the UE 201 and the EPC 203, and provides bearer and connection management. The HSS 2032 is used to provide some registers to manage functions such as home location register (not shown in the figure), and save some user-specific information about service features, data rates, etc. All user data can be transmitted through the SGW 2034, the PGW 2035 can provide IP address allocation and other functions for the UE 201, and the PCRF 2036 is a policy and charging control policy decision point for service data flow and IP bearer resources, which selects and provides available policy and charging control decisions for policy and charging execution function units (not shown in the figure).

[0086] The IP service 204 can include the Internet, an intranet, an IMS (IP Multimedia Subsystem), or other IP services.

[0087] Although the above describes the LTE system as an example, those skilled in the art should know that the present application is not only applicable to the LTE system, but also applicable to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA, 5G, and future new network systems (such as 6G), etc., which are not limited here.

[0088] Based on the above mobile terminal hardware structure and communication network system, various embodiments of the present application are proposed.

[0089] First embodiment

[0090] Referring to FIG. 3, which is a flowchart of an image processing method according to the first embodiment, the image processing method of the present embodiment can be applied to a processing device, including the steps of:

[0091] S10, filtering at least one image block using a neural network comprising at least one partial convolution module to obtain a target image block.

[0092] In the present embodiment, the processing device can be a smart terminal such as a mobile phone, a computer, etc., or a server such as a local server or a cloud server. In the present embodiment and the present application, the processing device is mainly illustrated as a smart terminal.

[0093] Optionally, the technical solution of the present embodiment can be applied to the fields of image coding, video coding, hardware video coding, special circuit video coding, real-time video coding, etc.

[0094] Optionally, the partial convolution module can perform partial convolution processing on the features of the at least one image block. The target image block can be an image block after filtering processing. The filtering processing can be loop filtering processing or post-filtering processing. At the decoding end, the target image block can be a decoded image block to be output and displayed. At the encoding end, the target image block can be a reference image block. The neural network can be a low-complexity neural network loop filter based on partial convolution and reparameterization, or the neural network can be a low-complexity neural network post-filter based on partial convolution and reparameterization.

[0095] Optionally, for the convenience of understanding, at least one neural network used in loop filtering in the coding process is briefly introduced.

[0096] Optionally, a LC-NNLF (Low Complexity Neural Network based Loop Filter, low complexity neural network loop filter) structure design is introduced in NNVC (Neural Network based Video Coding, neural network video compression).

[0097] Optionally, this neural network is used as the original neural network. As shown in FIG4 , the original neural network input includes at least one of an image block, a boundary strength, and a quantization parameter. For example, it may include an NxN YUV image block, a boundary strength BS (Boundary Strength), and a quantization parameter, such as a quantization step size QS (Qstep). Optionally, the network input layer of the original neural network includes a convolutional layer, which may be a 3x3 convolutional layer with a shape of 3x3x10xM, which is where the 3x3 conv 3x3x10xM in the figure is located. It may also include an activation function, such as a Leaky ReLu function.

[0098] Optionally, at least one hidden layer (i.e., n Hidden layers) is set in the original neural network, and the network architecture of each hidden layer can be the same or different, for example, it can include a 1x1 convolutional layer, such as the position of 1x1 conv 1x1xMxM in the figure. Optionally, it can also include an activation function, such as a LeaKy ReLu function. Optionally, it can also include a 1x1 convolutional layer, such as the position of 1x1 conv 1x1xMxK in the figure. Optionally, it can also include a 3x3 convolutional layer, such as the position of 3x3 conv 3x3xKxK in the figure.

[0099] Optionally, CP decomposition can be performed in the 3x3 convolutional layer, including four convolutional layers, namely, a 1x1 convolutional layer with a shape of 1x1xKxR, such as the position of 1x1 conv 1x1xKxR in the figure; a convolutional layer corresponding to a separable convolution with a shape of 3x1xRxR, such as the position of 3x1 Sep conv 3x1xRxR in the figure; a convolutional layer corresponding to a separable convolution with a shape of 1x3xRxR, such as the position of 1x3 Sep conv 1x3xRxR in the figure; and a convolutional layer corresponding to a convolution with a shape of 1x1xRxK, such as the position of 1x1 conv 1x1xRxK in the figure.

[0100] Optionally, two 1x1 convolutional layers can be fused, for example, the corresponding position area of ​​1x1 conv 1x1xKxR in the figure and the corresponding position area of ​​1x1 conv 1x1xMxK in the figure are convolutionally fused to obtain a new 1x1 convolutional layer with a shape of 1x1xMxR.

[0101] Optionally, two 1x1 convolutional layers can be fused, for example, the position of 1x1 conv 1x1xRxK and the position of 1x1 conv 1x1xKxM in the figure are convolutionally fused to obtain another new 1x1 convolutional layer with a shape of 1x1xRxM.

[0102] Optionally, the network output layer connected with the hidden layer comprises a convolution layer, which can be a 3x3 convolution layer, and the position of the 3x3 convolution layer can refer to the position of 3x3 conv 3x3xKxL in the figure.

[0103] Optionally, the output part comprises at least one image block after filtering processing.

[0104] Optionally, the boundary strength BS can be a measure for quantifying the sharpness or strength of the edge between adjacent image blocks.

[0105] Optionally, the quantization step QS can be a parameter for controlling the precision of quantization calculated according to the quantization parameter QP.

[0106] Optionally, the quantization step can determine the degree of precision loss in the process of converting the original signal (such as pixel value or transform coefficient) from continuous analog domain to discrete digital domain. The quantization process is to map continuous data to a finite number of discrete values, and to achieve data compression by discarding part of the details. The larger the quantization step, the greater the difference between each quantization level, and therefore the more details discarded, which will lead to an increase in compression efficiency, but also introduce more distortion, i.e. image quality degradation. Conversely, the smaller the quantization step, the more details retained, and the higher the image quality, but the required storage space and transmission bandwidth also increase accordingly.

[0107] Optionally, the input size of the NNVC filtering process is 144x144, which includes the current CTU (Coding Tree Unit) and 8 adjacent samples on each side. The luma samples are interleaved into four 72x72 blocks before being used as the input of the filtering process. The output tensor corresponds to the filtered CTU samples, organized into 64x64 blocks, including 4 luma blocks and 2 chroma blocks. Optionally, the QP (Quantizer Parameter) and BS (Boundary Strength) information are used as additional input information of the network.

[0108] Optionally, in the network architecture of the original neural network, the network input layer includes a 3x3 convolution layer and an activation function, which can accept samples with 10 channel inputs, and the output feature dimension is M=72. Optionally, the network architecture also includes n=11 hidden layers. For each hidden layer, the input features first pass through a 1x1 convolution layer, the output feature dimension is M=72, then pass through an activation function, and then the features pass through a second 1x1 convolution layer, the output feature dimension is reduced to k=24. Finally, a separable 3x3 convolution layer combined with CP decomposition is used. Optionally, the separable 3x3 convolution layer combined with CP decomposition can be composed of four convolution layers, and the rank R of the decomposition is set: the first layer includes a 1x1xKxR convolution; the second layer includes a 3x1xRxR separable convolution; the third layer includes a 1x3xRxR separable convolution; the fourth layer includes a 1x1xRxK convolution. Optionally, the 1x1 convolution of the first layer and the adjacent 1x1 convolution can be fused to reduce the LC-NNLF complexity. The final output layer contains a 3x3 convolution layer, which outputs 6 filtered samples of features for final residual scaling, such as 4 luminance and 2 chrominance feature filtered samples.

[0109] Optionally, due to the following defects in the original neural network architecture: the CP decomposition step sometimes produces an initial value with an unusually large absolute value after weight decomposition, which causes instability in the subsequent training phase of the network, and further causes the network to not converge. In the model corresponding to the original neural network, a larger number of convolution layer features is usually set to improve the performance of the model, but this will cause the number of features to be too large, resulting in model redundancy, i.e., the complexity of the model and the complexity of video encoding and / or decoding are not effectively reduced.

[0110] Optionally, due to the instability of the LC-NNLF, the neural network or neural network model does not converge, which limits the reproducibility of the neural network and increases the difficulty of network training, and / or affects the image quality enhancement effect in the filtering process during encoding and decoding. Optionally, due to the inconsistent decomposition of weights by CP decomposition each time, an unstable initial value is generated. Optionally, the original filter uses CP decomposition to reduce network complexity, which can reduce network complexity to a certain extent, but does not fully consider the redundancy of the number of convolution layer features.

[0111] Optionally, to avoid the above-mentioned defects, the network architecture of the above-mentioned neural network can be adjusted and set to construct a neural network containing at least one partial convolution module. That is, a low-complexity NNLF is constructed without CP decomposition and fully considering the feature redundancy of the convolution layer, thereby further reducing the complexity of encoding and decoding.

[0112] Optionally, the LC-NNLF can be determined, which is stable in the training process and considers channel redundancy, to stabilize the enhancement of loop filtering images and reduce the time overhead of the processing stage. Optionally, the original CP decomposition module is replaced by a partial convolution strategy, the convolution feature redundancy is fully considered by adjusting the partial convolution coefficient, the network redundancy is effectively reduced, and the complexity of the neural network loop filter in the loop filtering stage is further reduced, thereby reducing the complexity of the coding and decoding.

[0113] Optionally, by introducing the reparameterization method, the performance of the LC-NNLF is improved without increasing the complexity of the network, and the enhancement effect of the image on the Y channel is improved in the loop stage, and the enhancement effect of the image on the U and V channels is also improved. Further, the model architecture of the neural network containing at least one partial convolution module is stable in the training stage, and the phenomenon of unstable performance enhancement in the decoding loop filtering stage is avoided. By considering the convolution feature redundancy, the complexity of the LC-NNLF is effectively reduced, thereby reducing the computational complexity of the video coding loop filtering stage.

[0114] Optionally, the neural network containing at least one partial convolution module in the embodiment can be a low-complexity neural network loop filter, and the architecture can be as shown in FIG. 5. The input part includes at least one of an image block, a boundary strength, and a quantization parameter. For example, it can include an NxN YUV image block, which can be Yx4+U+V (72x72x10), and can include a boundary strength BS (such as BS info), and can include a quantization parameter such as a quantization step QStep. The network input layer of the neural network containing at least one partial convolution module includes a convolution layer, which can be a 3x3 convolution layer with a shape of 3x3x10xM, that is, the position of 3x3 conv 3x3x10xM in the figure. It can also include an activation function such as a LeaKy ReLu function. Optionally, at least one hidden layer is set in the neural network, such as n Hidden layers (i.e., n hidden layers, n is any natural number, such as 11, etc.). A partial convolution module can be set in each hidden layer, and the area module corresponding to the position of 3x3 conv 3x3xKxK in the figure.

[0115] Optionally, the architecture of each hidden layer can be the same or different, such as including a 1x1 convolution layer, i.e., 1x1 conv 1x1xMxK in the figure. Optionally, it can also include a 1x1 convolution layer, i.e., 1x1 conv 1x1xKxM in the figure. Optionally, it can also include an activation function such as a LeaKy ReLu function.

[0116] Optionally, the partial convolution module in the training phase and the partial convolution module in the recommendation phase of the neural network can be different.

[0117] Optionally, the channel division can be performed on the at least one image block feature to obtain a first sub-feature set including K / α channel numbers and a first sub-feature set including K*(1-1 / α) channel numbers. Optionally, in the training phase, the process corresponding to the Training phase in the figure can be performed, that is, the first sub-feature set including K*(1-1 / α) channel numbers is input into a branch with an identity mapping function. The first sub-feature set including K / α channel numbers is input into a branch corresponding to a plurality of convolution layers for convolution processing to obtain a second sub-feature set, and the outputs of the two branches are fused.

[0118] Optionally, a plurality of convolution layers can be arranged in the branch corresponding to the plurality of convolution layers, such as a 3x3 convolution layer and a 1x1 convolution layer. Optionally, the position of the 3x3 convolution layer in the branch can refer to the position of 3x3 conv 3x3xKx(K / α)*β in the figure. The position of the 1x1 convolution layer in the branch can refer to the position of 1x1 conv 1x1x(K / α)*βx(K / α) in the figure. Optionally, a function Concat is arranged in the partial convolution module to fuse the outputs of the two branches.

[0119] Optionally, in the application phase (that is, the inference phase), the process corresponding to the Inference phase in the figure can be performed, that is, the first sub-feature set including K*(1-1 / α) channel numbers is input into a branch with an identity mapping function. The first sub-feature set including K / α channel numbers is input into a branch including at least one convolution layer for convolution processing to obtain a second sub-feature set, and the outputs of the two branches are fused. At this time, there is one convolution layer, such as a 3x3 convolution layer, in the branch including at least one convolution layer. Optionally, the position of the 3x3 convolution layer in the branch can refer to the position of 3x3 conv 3x3xKx(K / α) in the figure. Optionally, a function Concat is arranged in the partial convolution module to fuse the outputs of the two branches. Optionally, the 3x3 convolution layer in the inference phase can be obtained by fusing the 3x3 convolution layer and the 1x1 convolution layer in the training phase.

[0120] Optionally, the network output layer connected with the hidden layer includes one convolution layer, which can be a 3x3 convolution layer, such as the position corresponding to 3x3 conv 3x3xKxL in the figure. Optionally, the output part includes at least one image block after filtering processing.

[0121] Optionally, two convolutional layers in the network input layer and the network output layer of the neural network can be set differently in the training phase and the inference phase. That is, a plurality of convolutional layers are set for training in the training phase, the plurality of convolutional layers trained are fused to form a new convolutional layer in the inference phase, and the new convolutional layer is used for actual inference application.

[0122] Optionally, a 3x3 convolutional layer with a shape of 3x3xKxL in the network output layer is exemplified. In the training phase, the process corresponding to the Training phase in the figure can be performed, that is, a 3x3 convolutional layer with a shape of 3x3xKx(k*β) and a 1x1 convolutional layer with a shape of 1x1x(k*β)xL are used. In the inference phase, the process corresponding to the Inference phase in the figure can be performed, that is, a 3x3 convolutional layer with a shape of 3x3xKxL is used.

[0123] Optionally, in this embodiment, the partial convolution coefficient α can be set according to the required degree of reduction of complexity, and then the number of channels participating in convolution operation in the neural network is controlled according to the partial convolution coefficient α, so as to achieve the purpose of reducing the complexity of the neural network.

[0124] Optionally, for an image block feature with K channels, the image block feature is divided into at least two first feature sets according to the partial convolution coefficient α, and the number of channels of the first feature sets is K / α and K / (1-1 / α) respectively. Optionally, in the partial convolution module of the neural network, only the first feature set with the number of channels of K / α can be subjected to convolution operation to obtain a second feature set, and the second feature set is spliced with the first feature set with the number of channels of K / (1-1 / α) to obtain the output of the partial convolution module.

[0125] Optionally, in order to fully fuse the two feature sets after splicing and not to increase the complexity of the filter, the second 1x1 convolutional layer of each hidden layer in the neural network can be moved to the position after the partial convolution module to serve as a feature fusion function, and the position of the first 1x1 convolutional layer in the hidden layer is kept unchanged, and the activation function is adjusted to the last position of the hidden layer to enable the neural network to obtain nonlinear capability.

[0126] Optionally, in the training phase of the neural network, the partial convolution module in the neural network can be widened or deepened by using the dilation coefficient β, and the corresponding processing is performed by using the reparameterization method.

[0127] Optionally, the original 3x3 convolutional layer is expanded into a 3x3 convolutional layer and a 1x1 convolutional layer. In the inference phase, the expanded 3x3 convolutional layer and the 1x1 convolutional layer are fused into a new 3x3 convolutional layer, so as to improve the performance of the neural network without increasing the complexity.

[0128] Optionally, for a given input image or image block X, X∈R C×H×W , where C is the number of channels, H is the image height, and W is the image width. The dimension parameter information of the two convolution kernels is and respectively, where and represent the input and output channel numbers of the convolution layer respectively, and k1 and k2 represent the convolution kernel size respectively. The convolution process can be represented as: X out =(X*W1)*W2;

[0129] Optionally, * represents the convolution process. According to the convolution property, the above process can be re-expressed as: X out =X*(W1*W2)=X*W new ;

[0130] where W new is the new convolution kernel after fusion. Through the above transformation, a new convolution kernel can be used to replace the previous two convolution kernels to complete the re-parameterization process.

[0131] Optionally, when training the network model of the neural network, two model training can be performed for high QP and low QP to obtain a neural network suitable for high QP and a neural network suitable for low QP. Optionally, the two neural networks each contain at least one convolution module.

[0132] Optionally, the data set and training data can be prepared according to the guide manual of the video standard research reference software. Optionally, the images in the data set are converted into YUV format to obtain original data; then, the training GT is generated according to the five QP values (22, 27, 32, 37, 42) respectively corresponding to the reduction of 2 (20, 25, 30, 35, 40) as the QP range of the generated data; and the obtained YUV original data is encoded and decoded by using the codec in VTM to generate the compressed data required for training, and then the training data QP is determined or obtained.

[0133] Optionally, the training data QP can be divided to train at least two neural networks. Optionally, the two neural networks to be trained can be model0 and model1 respectively. Optionally, model0 can use low QP (20, 25, 30) data for model training. Model1 can use high QP (35, 40, 45) data for training. Optionally, using high QP (35, 40, 45) data for training model1, such as 100 epochs (iteration times), the baseline model weight is obtained, and the training loss function has a penalty coefficient of 4 for the luminance component and a penalty coefficient of 1 for the chrominance component.

[0134] Optionally, model1 can load the baseline model weight to obtain the final model1 by iterating 400 epochs. Optionally, in order to balance the performance difference between the luminance and chrominance components, the loss function used in the first 150 epochs has a penalty coefficient of 4 for the luminance component, and in the last 250 epochs, the penalty coefficient for the luminance component is increased to 12, so as to force the neural network to pay more attention to the enhancement ability of the luminance component. The penalty coefficient for the color component is set to 1 throughout the training process of the neural network.

[0135] Optionally, model0 model also loads the baseline model weight and uses low QP (20, 25, 30) data for training. The final model0 is obtained by iterating 400 epochs. Optionally, in order to balance the performance difference between the luminance and chrominance components, the loss function used in the first 150 epochs has a penalty coefficient of 4 for the luminance component, and in the last 250 epochs, the penalty coefficient for the luminance component is increased to 12, so as to force the neural network to pay more attention to the enhancement ability of the luminance component. The penalty coefficient for the color component is set to 1 throughout the training process of the neural network.

[0136] Optionally, after obtaining the trained model1 and model0, the same operation is performed on the two neural networks to obtain the neural network in the inference stage. That is, the 3x3 convolution and 1x1 convolution extended in the training stage in each part of the convolution module of the neural network are merged, and the first 3x3 convolution kernel and the second 1x1 convolution kernel weight coefficient are multiplied by matrix layer by layer to obtain the final fused 3x3 convolution, that is, the part of the convolution module in the inference stage.

[0137] Optionally, the tensorflow can be converted into an onnx (Open Neural Network Exchange) model by using a tool provided by tensorflow, and the onnx model can be further converted into an SADL (Service Adaptation Description Language) model.

[0138] Optionally, after the network model training of the neural network is completed and the performance of the neural network reaches the optimal, the neural network can be used for filtering processing in the loop filtering stage of the coding and decoding process. For example, at least one image block is input into the neural network including at least one partial convolution module for model training, so as to realize filtering processing of the at least one image block, and a target image block is obtained according to the processing result of the filtering processing.

[0139] Optionally, if at the decoding end, the target image block can be a decoded image block.

[0140] Optionally, if at the encoding end, the target image block can be an image block after filtering processing.

[0141] Optionally, for the convenience of understanding, the coding and decoding process is briefly introduced. As shown in FIG. 6, the coding and decoding process includes a general coding control, a transform and quantization, an intra estimation, an intra prediction, a motion compensation, a motion estimation, an inverse quantization and inverse transform, a filter control analysis, a deblocking filter and an SAO filter (i.e., loop filtering), an entropy coding, a decoding frame buffer and the like. Optionally, the motion compensation module can select intra / inter to determine the specific compensation. Optionally, when the entropy coding is performed, the general control data determined by the general coding control module, the variation quantization coefficient determined by the transform and quantization module, the intra prediction data and the filter control determined by the filter control analysis, and the motion data determined by the decoding frame buffer are used, and then the coding bit rate is obtained.

[0142] Optionally, the decoded video signal is output through the decoding frame buffer.

[0143] Optionally, the loop filtering can include a deblocking filter and an LC-NNLF, then the branch results are fused, and SAO (Sample Adaptive Offset) and ALF (Adaptive Loop Filter) processing are performed.

[0144] Optionally, since each coding unit is independently coded and quantized in the encoding process, it can cause obvious distortion and abrupt changes at the block boundary. The block filter can reduce or eliminate these visible blocking effects by smoothing the pixel values of the block boundary area, thereby improving the continuity and visual quality of the reconstructed video. Optionally, SAO can classify the reconstructed samples (such as the reconstructed blocks) and apply an offset to each class of samples to reduce coding noise and further improve image quality. The adaptive processing is based on the color difference, edge direction and other characteristics of the pixels. Optionally, ALF is a more advanced and flexible filtering method, which can adaptively select a suitable filter template for each small block (such as 4x4 pixels) to reduce coding distortion. The filtering decision of ALF is based on the direction and activity of local image gradient, which can better adapt to the changes of image content, thereby improving the quality of the decoded video, especially for removing coding noise and restoring fine details.

[0145] In the embodiment, at least one image block is filtered by a neural network comprising at least one partial convolution module to obtain a target image block. When filtering at least one image block by a neural network, the partial convolution module can be controlled to implement partial convolution, thereby reducing the complexity of the filter. And / or because the partial convolution module in the neural network is used for partial convolution processing, and the partial convolution module is controllable, the redundancy of the number of convolution features of the neural network can be fully considered, so that the redundancy of the neural network can be reduced when filtering by the neural network, thereby reducing the complexity of video encoding and / or decoding, and improving the efficiency of video encoding and / or decoding.

[0146] Second embodiment

[0147] Based on the first embodiment, a second embodiment is proposed.

[0148] In the embodiment, referring to FIG. 7, step S10 includes the following steps:

[0149] S11, input at least one of the at least one image block, the quantization parameter and the boundary strength to the neural network comprising at least one partial convolution module;

[0150] S12, performing convolution processing on the at least one image block according to the first convolution layer of the neural network to determine or obtain at least one image block feature to be input into the partial convolution module;

[0151] S13, performing convolution processing on the at least one image block feature according to the at least one partial convolution module to determine or obtain a target fusion feature;

[0152] S14, determining or obtaining at least one target image block after filtering processing according to the target fusion feature.

[0153] Optionally, when performing the filtering processing on the at least one image block, the processing device can determine at least one of the at least one image block, the quantization parameter and the boundary strength, and input the at least one of the at least one image block, the quantization parameter and the boundary strength into the neural network comprising the at least one partial convolution module, so as to perform the partial convolution processing on the at least one image block by the at least one partial convolution module. For example, the at least one image block, the quantization parameter and the boundary strength are input into the neural network comprising the at least one partial convolution module to perform corresponding model training, and the filtering processing is performed on the at least one image block in the neural network, and thus the at least one image block after the filtering processing is obtained.

[0154] Optionally, the image block after the filtering processing can be the target image block. Alternatively, the at least one image block after the filtering processing can be subjected to some corresponding processing, such as other types of filtering processing, and thus the at least one target image block is obtained.

[0155] Optionally, after the neural network receives the at least one image block, each part in the neural network, such as the input layer, the hidden layer and the like, performs corresponding processing in sequence to complete the filtering processing on the at least one image block.

[0156] Optionally, the input layer of the neural network comprises at least one convolution layer, and a convolution kernel of the convolution layer can be a 3x3 convolution kernel or a convolution kernel set according to other rules. Optionally, the first convolution layer in the input layer can be the first convolution layer of the neural network, and after the first convolution layer receives the image features corresponding to the at least one image block, the convolution processing can be performed, and the image features after the convolution processing are processed by using an activation function or the like to obtain the image features to be input into the hidden layer.

[0157] Optionally, at least one partial convolution module can be arranged in at least one hidden layer. For example, at least one partial convolution module can be arranged in each hidden layer. If there is at least one partial convolution module in the hidden layer connected with the input layer, the image block features to be input into the partial convolution module can be determined or obtained according to the image features to be input into the hidden layer.

[0158] Optionally, the at least one image block feature is input into the partial convolution module, and the at least one image block feature is subjected to the partial convolution processing in the partial convolution module to determine or obtain the target fusion feature.

[0159] Optionally, the at least one partial convolution processing can be performed in each hidden layer in sequence, and thus after the convolution processing is performed through all the hidden layers, the final target fusion feature is fused with the at least one image block originally input into the neural network to obtain the at least one target image block after the filtering processing.

[0160] Optionally, after inputting the at least one image block feature into the partial convolution module for corresponding processing, the target fusion feature can be obtained, and the target fusion feature can be further subjected to corresponding convolution processing and input into the partial convolution module in the next hidden layer for convolution processing to obtain the next target fusion feature, until all the partial convolution modules are subjected to convolution processing to obtain the final target fusion feature. Then, the at least one target image block subjected to filtering processing is determined or obtained according to the final target fusion feature and the at least one image block of the neural network, such as a decoded image block.

[0161] Optionally, the filtering processing procedures at the encoding end and the decoding end are similar, and steps S11-S14 can be performed.

[0162] Optionally, at the encoding end, the corresponding loop filtering parameters are generated when the encoding parameters are determined, such as the parameters required by the neural network, such as quantization parameters, boundary strength, and the like. These parameters reflect the best filtering strategy that the encoding end considers to be applied at the decoding end to optimize the reconstructed video quality, and these filtering parameters will be encoded into the bitstream.

[0163] Optionally, at the decoding end, the key frames, transform coefficients, quantization parameters, and various parameter information in the original bitstream are obtained by decoding the code stream, and after inverse quantization and inverse transformation are completed, the reconstructed image block without loop filtering processing is obtained, and the at least one reconstructed image block is subjected to filtering processing, such as filtering processing of the at least one image block by the neural network containing at least one partial convolution module to obtain the target image block.

[0164] Optionally, at the encoding end, the filtered image is used as a reference image for subsequent encoding motion compensation, and at the decoding end, the filtered image is output as a decoded image, or is subjected to filtering and then output. Optionally, the image includes at least one block, such as an image block.

[0165] Optionally, at least one image block, at least one of the quantization parameter and the boundary strength are input into the neural network comprising at least one partial convolution module, and the at least one image block feature is determined by performing convolution processing according to a first convolution layer of the neural network; the target fusion feature is determined or obtained by performing convolution processing on the at least one image block feature according to the at least one partial convolution module; and the at least one target image block after filtering processing is determined or obtained according to the target fusion feature. Thus, when the at least one image block is filtered by the neural network, the at least one image block feature is determined by the first convolution layer first, and then the at least one image block feature is partially convolved according to the at least one partial convolution module, so as to reduce the complexity of the filter. And since the partial convolution module in the neural network is used for partial convolution processing, and the partial convolution module is controllable, the redundancy of the convolution feature quantity of the neural network can be fully considered, and the number of feature channels of the convolution layer or the region of the image block feature is limited, so as to reduce the redundancy of the neural network when the neural network is used for filtering processing, thereby reducing the complexity of video encoding and / or decoding, and improving the efficiency of video encoding and / or decoding.

[0166] Optionally, step S14 comprises:

[0167] performing convolution processing on the target fusion feature according to a last convolution layer of the neural network;

[0168] determining or obtaining the at least one target image block after filtering processing according to the target fusion feature after convolution processing and the at least one image block input into the neural network.

[0169] Optionally, there can be multiple hidden layers in the neural network, and at least one partial convolution module can be arranged in each hidden layer, or no partial convolution module can be arranged.

[0170] Optionally, if a partial convolution module is arranged in the last hidden layer (i.e., the hidden layer adjacent to the output layer), after the image block feature of the at least one image block is convolved in the at least one hidden layer and enters the partial convolution module in the last hidden layer for partial convolution processing, the target fusion feature corresponding to the last partial convolution module is obtained, and then the convolution processing can be continued in the hidden layer by a convolution layer (such as a convolution layer corresponding to a 1x1 convolution kernel), and the processing is performed by an activation function, and then the at least one target image block after filtering processing is determined or obtained by combining processing with the at least one image block input into the neural network.

[0171] Optionally, if no partial convolution module is set in the last hidden layer, the last hidden layer in which the last partial convolution module is located is determined, and the target fusion feature corresponding to the last partial convolution module is obtained in the hidden layer, and the target fusion feature is processed through the remaining hidden layers, and then combined with the at least one image block input into the neural network, thereby determining or obtaining the at least one target image block after filtering processing.

[0172] Optionally, after obtaining the target fusion feature by performing partial convolution on the at least one image block feature according to the partial convolution module, the last convolution layer of the neural network can be further used for convolution processing, and then combined with the at least one image block input into the neural network, thereby determining or obtaining the at least one target image block after filtering processing, and further ensuring the effectiveness of the obtained at least one target image block, and improving the efficiency of video encoding and / or decoding.

[0173] By the technical solutions of the embodiment, the redundancy of the neural network can be reduced when the neural network is used for filtering processing, thereby reducing the complexity of video encoding and / or decoding, and further improving the efficiency of video encoding and / or decoding.

[0174] Third embodiment

[0175] Based on the first or second embodiment, the third embodiment is proposed.

[0176] In the embodiment, step S13 comprises:

[0177] Inputting the at least one image block feature into at least one partial convolution module;

[0178] Dividing the at least one image block feature by channel or region to determine or obtain at least two first sub-feature sets of the image block;

[0179] Performing convolution processing on the at least one first sub-feature set to determine or obtain a second sub-feature set;

[0180] Determining or obtaining a target fusion feature according to the second sub-feature set and the first sub-feature set which has not been processed by convolution.

[0181] Optionally, if at least one partial convolution module is set in at least one hidden layer, each partial convolution module performs the same operation on the image block feature, i.e., partial convolution processing.

[0182] Optionally, in the neural network, the at least one image block feature of the at least one image block is input into at least one partial convolution module, and the at least one image block feature is divided in the partial convolution module.

[0183] Optionally, the at least one image block feature is divided by channel, and then at least two channel numbers of image block features are determined or obtained, and the at least two channel numbers of image block features are taken as at least two first feature sets.

[0184] Optionally, the two channel numbers can be the same or different, such as 10 channels being divided into two 5 channels, or being divided into 7 channels and 3 channels.

[0185] Optionally, the at least one image block is divided by region, and image block features corresponding to different regions are obtained, and the image block features corresponding to each region are taken as a first feature set.

[0186] Optionally, the at least one image block feature is divided according to the distribution of different pixels in the at least one image block, and then at least two first feature sets of the at least one image block are obtained.

[0187] Optionally, the at least one image block feature is divided according to a historical division rule, and at least two first feature sets of the image block are determined or obtained.

[0188] Optionally, the first feature set includes at least one first feature, and the first feature can be an image block feature.

[0189] Optionally, the number of first feature sets is at least two, and can also be multiple. In this embodiment, only two first feature sets are taken as examples, but when the partial convolution module is partially convolved in this embodiment, it is not limited to the two first feature sets.

[0190] Optionally, if the number of first feature sets is two, one of the two first feature sets can be selected for convolution processing.

[0191] Optionally, the selection rule of the first feature set can be to select the first feature set with the least number of channels, or to randomly select a first feature set, or to select based on a certain rule, such as determining that a first feature set corresponding to a certain region of the image block needs to be convolved according to a preset rule, and then the first feature set can be taken as the selected first feature set.

[0192] Optionally, after at least one first feature set is selected, the selected at least one first feature set can be convolved in the partial convolution module, and the first feature set after convolution is taken as a second feature set.

[0193] Optionally, the second feature set obtained by convolution and the first feature set without convolution can be fused, and then a target fusion feature is obtained.

[0194] Optionally, the target fusion feature comprises image block features in the second sub-feature set and image block features in the first sub-feature set.

[0195] Optionally, when performing the partial convolution processing according to the partial convolution module, the at least one image block feature input into the partial convolution module can be divided by channel or region to determine or obtain at least two first sub-feature sets, the at least one first sub-feature set is subjected to convolution processing, and then a second sub-feature set is obtained, and the target fusion feature is determined or obtained according to the second sub-feature set and the first sub-feature set which is not subjected to convolution processing. Thus, the partial convolution module can be used to perform partial convolution on the image block features of at least one image block in the neural network. Moreover, the partial convolution module is controllable, and can fully consider the redundancy of the number of convolution features in the neural network, limit the number of feature channels of the convolution layer or the region of the image block features, and thus can reduce the redundancy of the neural network when the neural network is used for filtering processing, thereby reducing the complexity of video encoding and / or decoding, and improving the efficiency of video encoding and / or decoding.

[0196] Optionally, the division of the at least one image block feature by channel or region comprises at least one of the following mode one to mode three:

[0197] Mode one, the at least one image block feature is divided by channel or region according to the parameters of the partial convolution module;

[0198] Optionally, the parameters of the partial convolution module can be parameters agreed in advance by the encoding end and the decoding end, such as the number of divided channels, or the divided region, or the number of divisions.

[0199] Optionally, after the parameters of the partial convolution module are determined or obtained, the type of division of the at least one image block feature can be determined according to the parameters of the partial convolution module. If the at least one image block feature needs to be divided by channel, the at least one image block feature can be divided by channel, and the first sub-feature set is determined according to the division result.

[0200] Optionally, the number of channels corresponding to the image block features in each first sub-feature set can be different or the same. If the at least one image block feature needs to be divided by region, the at least one image block feature can be divided by channel, and the first sub-feature set is determined according to the division result.

[0201] Optionally, the regions of the image blocks corresponding to the image block features in each first sub-feature set are different.

[0202] Optionally, when the at least one image block feature is processed by the partial convolution module, the at least one image block feature can be divided by channel or region according to the parameters of the partial convolution module, so as to ensure the effectiveness of the division and facilitate subsequent partial convolution processing by the partial convolution module.

[0203] Optionally, the partial convolution module is controllable, can fully consider the redundancy of the number of convolution features of the neural network, limit the number of feature channels of the convolution layer or the region of the image block feature, and thus can reduce the redundancy of the neural network when the neural network is used for filtering processing, thereby reducing the complexity of video encoding and / or decoding, and improving the efficiency of video encoding and / or decoding.

[0204] Optionally, the at least one image block feature can be divided by channel according to the number of channels corresponding to at least one convolution layer in at least one hidden layer in the neural network.

[0205] Optionally, the number of channels corresponding to at least one convolution layer in at least one hidden layer in the neural network can be determined, and the at least one image block feature can be divided by channel according to the number of channels. For example, if the number of channels is 3, the at least one image block feature can be divided into image block features with two channels and image block features with one channel.

[0206] Optionally, the number of channels can be determined according to the number of convolution kernels in the partial convolution module in the hidden layer, and then the at least one image block feature can be divided by channel according to the number of channels.

[0207] Optionally, the number of channels can also be determined according to the at least one image block input to the neural network, and then the at least one image block feature can be divided by channel according to the number of channels.

[0208] Optionally, when the at least one image block feature is processed by the partial convolution module, the at least one image block feature can be divided by channel according to the number of channels corresponding to at least one convolution layer in at least one hidden layer in the neural network, so as to ensure the effectiveness of the division and facilitate subsequent partial convolution processing by the partial convolution module. Optionally, the partial convolution module is controllable, can fully consider the redundancy of the number of convolution features of the neural network, limit the number of feature channels of the convolution layer, and thus can reduce the redundancy of the neural network when the neural network is used for filtering processing, thereby reducing the complexity of video encoding and / or decoding, and improving the efficiency of video encoding and / or decoding.

[0209] Optionally, the at least one image block feature can be divided by region according to the image block segmentation flag determined or generated according to the image block feature.

[0210] Optionally, the image block segmentation flag can be determined or generated according to at least one image block feature of the at least one image block, and then the at least one image block feature is divided by region according to the image block segmentation flag.

[0211] Optionally, the image block segmentation flag can be a model parameter, a trigger instruction, or a division rule.

[0212] Optionally, when the at least one image block feature needs to be divided by region, the image block segmentation flag can be determined or generated according to the image block feature, or the image block segmentation flag can be determined according to a flag bit in a code stream. The image block feature corresponding to the image block segmentation flag in the image block is determined, and the image block feature is divided to obtain at least one first sub-feature set.

[0213] Optionally, the flag bit in the code stream can be 3 bits long, and the value is in the range of 000-111, which represents at most 8 division modes.

[0214] Optionally, in the partial convolution module, when the at least one image block feature is processed by partial convolution, the image block segmentation flag determined or generated according to the image block feature can be used to divide the at least one image block feature by channel, so as to guarantee the effectiveness of the division and facilitate subsequent partial convolution processing by the partial convolution module. Optionally, the partial convolution module is controllable, can fully consider the redundancy of the number of convolution features of the neural network, and can reduce the redundancy of the neural network when the neural network is used for filtering processing, thereby reducing the complexity of video encoding and / or decoding, and improving the efficiency of video encoding and / or decoding.

[0215] Through the technical solutions of the embodiment, the redundancy of the neural network can be reduced when the neural network is used for filtering processing, thereby reducing the complexity of video encoding and / or decoding, and improving the efficiency of video encoding and / or decoding.

[0216] Fourth embodiment

[0217] Based on any one of the first to third embodiments, the fourth embodiment is proposed.

[0218] In the embodiment, the image block features include chrominance component image block features and luminance component image block features.

[0219] Optionally, step S13 comprises:

[0220] The chrominance component image block features are processed by convolution by the partial convolution module corresponding to the chrominance component to determine or obtain the fusion features of the chrominance component;

[0221] The luminance component image block feature is convoluted by the partial convolution module corresponding to the luminance component to determine or obtain the fusion feature of the luminance component.

[0222] The target fusion feature is determined or obtained according to the fusion feature of the luminance component and the fusion feature of the chroma component.

[0223] Optionally, if the image block feature of the same image block includes the chroma component image block feature and the luminance component image block feature, the chroma component image block feature and the luminance component image block feature can be convoluted respectively in the partial convolution module of the neural network, and then fused to obtain the target fusion feature.

[0224] Optionally, at least three branches can be arranged in at least one partial convolution module, and the chroma component image block feature is determined and divided to obtain a plurality of chroma component image block sub-feature sets, and at least one chroma component image block sub-feature set is convoluted through one branch. The luminance component image block feature is determined and divided to obtain a plurality of luminance component image block sub-feature sets, and at least one luminance component image block sub-feature set is convoluted through another branch. The remaining branch is used to transmit at least one luminance component image block sub-feature set that has not been convoluted and / or at least one chroma component image block sub-feature set that has not been convoluted. Then, the at least one chroma component image block sub-feature set that has been convoluted, the at least one luminance component image block sub-feature set that has been convoluted, the at least one chroma component image block sub-feature set that has not been convoluted, and the at least one luminance component image block sub-feature set that has not been convoluted are fused to determine or obtain the target fusion feature.

[0225] Optionally, if different partial convolution modules are required to perform partial convolution on the luminance component image block feature and the chroma component image block feature respectively, the partial convolution module corresponding to the luminance component image block feature and the partial convolution module corresponding to the chroma component image block feature in the hidden layer of the neural network can be determined first. The chroma component image block feature is convoluted by the partial convolution module corresponding to the chroma component image block feature to determine or obtain the fusion feature of the chroma component. The luminance component image block feature is convoluted by the partial convolution module corresponding to the luminance component image block feature to determine or obtain the fusion feature of the luminance component.

[0226] Optionally, the convolution of the chroma component image block feature and the convolution of the luminance component image block feature can be performed simultaneously by multiple threads.

[0227] Optionally, for the chroma component image block feature, the chroma component image block feature can be divided by channel or region, and the division method can include at least one of the following:

[0228] According to the parameters of the partial convolution module corresponding to the chroma component image block feature, the chroma component image block feature is divided by channel or region;

[0229] According to the number of channels corresponding to at least one convolution layer in at least one hidden layer in the neural network, the chroma component image block feature is divided by channel;

[0230] According to the image block segmentation mark determined or generated by the chroma component image block feature, the chroma component image block feature is divided.

[0231] Optionally, in the partial convolution module corresponding to the chroma component image block feature, if the chroma component image block feature is divided by channel or region to obtain a plurality of chroma component image block sub-features, at least one chroma component image block sub-feature can be selected for convolution processing, and the at least one chroma component image block sub-feature after convolution processing is fused with the at least one chroma component image block sub-feature without convolution processing to determine or obtain the fusion feature of the chroma component.

[0232] Optionally, the at least one chroma component image block sub-feature without convolution processing can be the chroma component image block feature without convolution processing in the partial convolution module corresponding to the chroma component image block feature, and does not refer to other positions without convolution processing.

[0233] Optionally, for the luminance component image block feature, the luminance component image block feature can be divided by channel or region, and the division method can include at least one of the following:

[0234] According to the parameters of the partial convolution module corresponding to the luminance component image block feature, the luminance component image block feature is divided by channel or region;

[0235] According to the number of channels corresponding to at least one convolution layer in at least one hidden layer in the neural network, the luminance component image block feature is divided by channel;

[0236] According to the image block segmentation mark determined or generated by the luminance component image block feature, the luminance component image block feature is divided.

[0237] Optionally, in the partial convolution module corresponding to the luminance component image block feature, if the luminance component image block feature is divided by channel or region to obtain a plurality of luminance component image block sub-features, at least one luminance component image block sub-feature can be selected for convolution processing, and the at least one luminance component image block sub-feature after convolution processing and the at least one luminance component image block sub-feature without convolution processing are fused to determine or obtain the fusion feature of the luminance component. Optionally, the at least one luminance component image block sub-feature without convolution processing can be the luminance component image block sub-feature without convolution processing in the partial convolution module corresponding to the luminance component image block feature, and does not refer to the luminance component image block sub-feature without convolution processing in other positions.

[0238] Optionally, the fusion feature of the luminance component and the fusion feature of the chrominance component can be fused again to obtain the target fusion feature.

[0239] Optionally, if the image block features of the same image block include Y component image block features, U component image block features and V component image block features, the partial convolution module corresponding to the Y component image block features can be determined, and the Y component image block features are input into the partial convolution module corresponding to the Y component image block features, and the Y component image block features are divided in the partial convolution module to obtain Y component image block sub-features.

[0240] Optionally, the division rule can be at least one of the following:

[0241] The Y component image block features are divided by channel or region according to the parameters of the partial convolution module corresponding to the Y component image block features;

[0242] The Y component image block features are divided by channel according to the number of channels corresponding to at least one convolution layer in at least one hidden layer in the neural network;

[0243] The image block segmentation flag is determined or generated according to the Y component image block features, and the Y component image block features are divided by region according to the image block segmentation flag.

[0244] Optionally, at least one Y component image block sub-feature is selected for convolution processing, and the Y component image block sub-feature without convolution processing and the Y component image block sub-feature after convolution processing are fused to obtain the fusion feature of the Y component.

[0245] Optionally, the U component image block features are convolved according to the partial convolution module corresponding to the U component image block features, which is similar to the process of convolving the Y component image block features according to the partial convolution module corresponding to the Y component image block features, which will not be repeated here.

[0246] Optionally, the V-component image block features are convoluted by the partial convolution module corresponding to the V-component image block features, similar to the process of convoluting the Y-component image block features by the partial convolution module corresponding to the Y-component image block features, which will not be repeated here.

[0247] Optionally, after the fusion features of the Y component, the fusion features of the U component, and the fusion features of the V component are determined or obtained, the fusion features of the Y component, the fusion features of the U component, and the fusion features of the V component can be fused to determine or obtain target fusion features.

[0248] Optionally, if the image block features include R-component image block features, G-component image block features, and B-component image block features, the operation process steps of the partial convolution processing according to the partial convolution module are similar to those of the YUV components, which can be referred to the above process, and will not be repeated here. Optionally, when the image block features include chrominance component image block features and luminance component image block features, the chrominance component image block features can be convoluted by the partial convolution module corresponding to the chrominance component to determine or obtain fusion features of the chrominance component, and the luminance component image block features can be convoluted by the partial convolution module corresponding to the luminance component to determine or obtain fusion features of the luminance component, and then the target fusion features can be determined or obtained according to the fusion features of the luminance component and the fusion features of the chrominance component. Thus, when performing filtering processing, the differences between the chrominance component and the luminance component can be fully considered, so that the effect of the partial convolution module on the chrominance component image block features reaches the best, and the effect of the partial convolution module on the luminance component image block features reaches the best, and thus the performance of the target fusion features finally achieved is the best. Moreover, the partial convolution module is controllable, and can fully consider the redundancy of the neural network convolution features, so that the redundancy of the neural network can be reduced when the neural network is used for filtering processing, thereby reducing the complexity of video encoding and / or decoding, and improving the efficiency of video encoding and / or decoding.

[0249] By the technical solutions of the embodiment, the redundancy of the neural network can be reduced when the neural network is used for filtering processing, thereby reducing the complexity of video encoding and / or decoding, and improving the efficiency of video encoding and / or decoding.

[0250] Fifth Embodiment

[0251] Based on any one of the first to fourth embodiments, the fifth embodiment is proposed.

[0252] In the embodiment, the image processing method further includes at least one of the following:

[0253] The fourth mode is to use the partial convolution module to perform convolution processing on the image block features in the partial channels.

[0254] Optionally, at least one hidden layer can be present in the neural network, and at least one partial convolution module can be arranged in each hidden layer. Optionally, the partial convolution module can be arranged between two convolution layers in the hidden layer, such as between two 1x1 convolution kernels.

[0255] Optionally, in all channels corresponding to the at least one partial convolution module, convolution processing can be performed on the image block features in the partial channels, so that the convolution processing of the image block features is completed in the partial channels, and the convolution processing of the image block features is not performed in the other partial channels. That is, the partial convolution processing of the image block features is realized.

[0256] Optionally, the partial convolution module can be used to perform convolution processing on the image block features in the partial channels, so that the neural network only performs partial convolution on the at least one image block. Moreover, the partial convolution module is controllable, can fully consider the redundancy of the convolution feature quantity of the neural network, limit the number of feature channels of the convolution layer, and thus can reduce the redundancy of the neural network when the neural network is used for filtering processing, thereby reducing the complexity of video encoding and / or decoding, and improving the efficiency of video encoding and / or decoding.

[0257] The fifth mode is to use the partial convolution module to perform convolution processing on the partial regions of the image block features in the partial channels.

[0258] Optionally, the image block features corresponding to the at least one image block can be checked to determine whether there is a redundant region, and if there is a redundant region, only the non-redundant region is subjected to convolution processing, that is, the non-redundant region is taken as the partial region of the image block features.

[0259] Optionally, in all channels corresponding to the at least one partial convolution module, convolution processing can be performed on the image block features in the partial channels. Optionally, the number of channels corresponding to the partial convolution module can be determined according to the number of convolution kernels in the partial convolution module.

[0260] Optionally, after the image block features are input into the partial convolution module, convolution processing can be performed on the partial regions of the image block features in the partial channels, and the other partial channels do not perform convolution processing on the other partial regions of the image block features. The image block features subjected to convolution processing in the partial convolution module are fused with the image block features not subjected to convolution processing to obtain target fusion features.

[0261] Optionally, the partial convolution module can be used to perform convolution processing on part of the region of the image block feature on part of the channel, so as to realize that the neural network only performs partial convolution on the at least one image block. Moreover, the partial convolution module is controllable, can fully consider the redundancy of the convolution feature quantity of the neural network, limit the feature channel quantity of the convolution layer and the region of the image block feature, and thus can realize that the redundancy of the neural network is reduced when the neural network is used for filtering processing, so as to reduce the complexity of video encoding and / or decoding, and thus improve the efficiency of video encoding and / or decoding.

[0262] Optionally, the image block is a reconstruction of the block.

[0263] Optionally, if the processing device is a decoding end, the image block input into the neural network comprising at least one partial convolution module is a reconstruction of the block.

[0264] Optionally, in the decoding end, after the corresponding decoding operation is performed according to the data obtained from the code stream to obtain the reconstruction of the block, the reconstruction of the block can be input into the neural network comprising at least one partial convolution module to perform filtering processing, and the decoded image block is determined or obtained according to the reconstruction of the at least one block after the filtering processing, and the decoded image determined by the at least one decoded image block is output.

[0265] Optionally, since the image block can be a reconstruction of the block, the reconstruction of the block is filtered by using the at least one partial convolution module, so as to improve the efficiency of video decoding.

[0266] Optionally, the partial convolution module comprises at least one branch having an identity mapping function.

[0267] Optionally, at least one partial convolution module in the hidden layer in the neural network can be provided with at least one branch having an identity mapping function. If the image block feature passes through the branch, no convolution processing is performed, that is, the input feature and the output feature corresponding to the branch having the identity mapping function can be consistent.

[0268] Optionally, at least one branch having an identity mapping function can be provided in the partial convolution module, so as to realize that no convolution processing is performed when the image block feature passes through the branch, to realize the partial convolution function, and thus the redundancy of the neural network can be reduced when the neural network is used for filtering processing, so as to reduce the complexity of video encoding and / or decoding, and thus improve the efficiency of video encoding and / or decoding.

[0269] Optionally, the partial convolution module comprises at least one branch of a convolution layer.

[0270] Optionally, at least one of the convolution modules in the hidden layer in the neural network can be provided with a branch including at least one convolution layer, through which the image block features can be convoluted.

[0271] Optionally, the branch with the identity mapping function and the branch including at least one convolution layer can be simultaneously provided in the partial convolution module.

[0272] Optionally, after the at least one image block feature is input into the partial convolution module, the at least one image block feature is divided in the partial convolution module, and at least two first sub-feature sets of the image block are determined or obtained, selection can be performed in the two first sub-feature sets, and the selected first sub-feature set is input into the branch including at least one convolution layer, and the first sub-feature set is convoluted by the convolution layer in the branch to obtain a second sub-feature set. The other first sub-feature set is input into the branch with the identity mapping function, and the outputs of the branch with the identity mapping function and the branch including at least one convolution layer are merged to fuse the first sub-feature set output by the branch with the identity mapping function without convolution processing and the first sub-feature set output by the branch including at least one convolution layer without convolution processing, thereby obtaining a target fusion feature.

[0273] Optionally, the branch including at least one convolution layer can be provided in the partial convolution module, so that the image block features can be normally convoluted when passing through the branch, and the partial convolution function of the partial convolution module can be effectively realized when combined with the branch with the identity mapping function, thereby reducing the redundancy of the neural network when the neural network is used for filtering processing, so as to reduce the complexity of video encoding and / or decoding, thereby improving the efficiency of video encoding and / or decoding.

[0274] Optionally, the image processing method further includes at least one of the following:

[0275] Optionally, the number of partial convolution modules in the neural network is determined according to the flag bit.

[0276] Optionally, when the processing device is a decoding end, the flag bit can be obtained from a bitstream. Optionally, the flag bit can include a flag bit corresponding to the neural network and a flag bit corresponding to the partial convolution module in the neural network.

[0277] Optionally, the neural network to be used can be determined according to the corresponding flag bit in the preset plurality of neural networks, and the number of partial convolution modules in the neural network to be used can be determined according to the corresponding flag bit of the partial convolution module in the neural network, such as the number of partial convolution modules used in each hidden layer. And determine the finally used neural network containing at least one partial convolution module, input the reconstructed block into the neural network for filtering processing to obtain the target image block.

[0278] Optionally, the number of partial convolution modules in the neural network can be determined according to the flag bit at the decoding end, thereby ensuring the effectiveness of the obtained neural network, facilitating subsequent filtering processing according to the neural network, reducing the redundancy of the neural network, thereby reducing the complexity of video encoding and / or decoding, and thereby improving the efficiency of video encoding and / or decoding.

[0279] Method ten, determining the number of branches in the partial convolution module in the neural network according to the flag bit;

[0280] Optionally, when the processing device is a decoding end, and the neural network to be used and the number of partial convolution modules to be used in each hidden layer of the neural network have been determined in the decoding end. The flag bit can be obtained in the code stream, and the corresponding partial convolution module in the neural network and the number of branches in the partial convolution module can be determined according to the flag bit, such as including two branches, respectively, a branch with an identity mapping function and a branch including at least one convolution layer.

[0281] Optionally, the number of branches in the partial convolution module in the neural network can be determined according to the flag bit at the decoding end, thereby ensuring the effectiveness of the obtained neural network, facilitating subsequent filtering processing according to the neural network, reducing the redundancy of the neural network, thereby reducing the complexity of video encoding and / or decoding, and thereby improving the efficiency of video encoding and / or decoding.

[0282] Method eleven, performing fusion processing on a plurality of convolution layers in the partial convolution module of the neural network in the training stage to determine or obtain the partial convolution module in the inference stage.

[0283] Optionally, after the processing device determines the neural network, the neural network can be trained first, and after the training is completed, the trained neural network can be applied to the filtering processing in the encoding and decoding process.

[0284] Optionally, when the neural network is trained, a plurality of convolution layers can be arranged in the branch of the at least one partial convolution module in the neural network, such as a convolution layer with a 3x3 convolution kernel and a convolution layer with a 1x1 convolution kernel. The neural network is trained, and after the training is completed, such as after the neural network converges, the plurality of convolution layers in the partial convolution module can be merged, that is, the convolution layer with a 3x3 convolution kernel and the convolution layer with a 1x1 convolution kernel are merged to form a new convolution layer with a 3x3 convolution kernel. The partial convolution module with the new convolution layer with a 3x3 convolution kernel is used as the partial convolution module in the inference stage. Optionally, the operation of step S10 can be performed in the inference stage.

[0285] Optionally, the plurality of convolution layers in the partial convolution module in the neural network in the training stage are fused to determine or obtain the partial convolution module in the inference stage, so that the performance of the neural network is optimized during training in the training stage, and the complexity of the neural network is reduced by fusing too many convolution layers in the inference stage, thereby reducing the redundancy of the neural network during filtering processing based on the neural network, thereby reducing the complexity of video encoding and / or decoding, and further improving the efficiency of video encoding and / or decoding.

[0286] According to the technical solutions of the embodiments, the redundancy of the neural network can be reduced during filtering processing based on the neural network, thereby reducing the complexity of video encoding and / or decoding, and further improving the efficiency of video encoding and / or decoding.

[0287] Sixth embodiment

[0288] The embodiments of the present application also provide a processing device, referring to FIG. 8, the processing device comprises:

[0289] The processing module A10 is configured to perform filtering processing on the at least one image block by using the neural network comprising the at least one partial convolution module to obtain the target image block.

[0290] Optionally, the processing module A10 comprises:

[0291] The input unit is configured to input at least one of the at least one image block, the quantization parameter and the boundary strength to the neural network comprising the at least one partial convolution module;

[0292] The convolution unit is configured to perform convolution processing on the at least one image block according to the first convolution layer of the neural network to determine or obtain the at least one image block feature to be input into the partial convolution module;

[0293] The fusion unit is configured to perform convolution processing on the at least one image block feature according to the at least one partial convolution module to determine or obtain the target fusion feature;

[0294] The determining unit is configured to determine or obtain the target fusion feature according to the target image block after filtering.

[0295] Optionally, the fusion unit is further configured to:

[0296] input the image block feature into the at least one partial convolution module;

[0297] divide the image block feature by channel or region to determine or obtain at least two first sub-feature sets of the image block;

[0298] perform convolution processing on the at least one first sub-feature set to determine or obtain a second sub-feature set;

[0299] determine or obtain the target fusion feature according to the second sub-feature set and the first sub-feature set that has not been subjected to convolution processing.

[0300] Optionally, the dividing the image block feature by channel or region comprises at least one of:

[0301] dividing the image block feature by channel or region according to parameters of the partial convolution module;

[0302] dividing the image block feature by channel according to a number of channels corresponding to at least one convolution layer in at least one hidden layer in the neural network;

[0303] determining or generating an image block segmentation mark according to the image block feature, and dividing the image block feature by region according to the image block segmentation mark.

[0304] Optionally, the image block feature comprises a chrominance component image block feature and a luminance component image block feature.

[0305] Optionally, the fusion unit is further configured to:

[0306] perform convolution processing on the chrominance component image block feature according to a partial convolution module corresponding to the chrominance component to determine or obtain a fusion feature of the chrominance component;

[0307] perform convolution processing on the luminance component image block feature according to a partial convolution module corresponding to the luminance component to determine or obtain a fusion feature of the luminance component;

[0308] determine or obtain the target fusion feature according to the fusion feature of the luminance component and the fusion feature of the chrominance component.

[0309] Optionally, the determining unit is further configured to:

[0310] perform convolution processing on the target fusion feature according to a last convolution layer of the neural network;

[0311] According to the target fusion feature after the convolution processing and the at least one image block input to the neural network, at least one target image block after the filtering processing is determined or obtained.

[0312] Optionally, the processing device further comprises at least one of the following:

[0313] The partial convolution module is used to perform convolution processing on the image block features on partial channels;

[0314] The partial convolution module is used to perform convolution processing on partial regions of the image block features on partial channels;

[0315] The image block is a reconstruction of the block;

[0316] The partial convolution module comprises at least one branch having an identity mapping function;

[0317] The partial convolution module comprises at least one convolution layer branch;

[0318] According to the flag, the number of partial convolution modules in the neural network is determined;

[0319] According to the flag, the number of branches in the partial convolution module in the neural network is determined;

[0320] The plurality of convolution layers in the partial convolution module in the neural network in the training stage are fused to determine or obtain the partial convolution module in the inference stage.

[0321] The processing device provided in the embodiments of the present application and the technical solutions shown in the corresponding method embodiments have similar implementation principles and beneficial effects, which will not be described here in detail.

[0322] The embodiments of the present application further provide a processing device, comprising a memory and a processor, the memory stores an image processing program, and the image processing program is executed by the processor to implement the steps of the image processing method in any of the above embodiments.

[0323] The embodiments of the present application further provide a storage medium, the storage medium stores an image processing program, and the image processing program is executed by the processor to implement the steps of the image processing method in any of the above embodiments.

[0324] In the embodiments of the intelligent terminal and the storage medium provided in the present application, any of the above image processing method embodiments can contain all the technical features, and the description and explanation content is basically the same as that of the above method embodiments, which will not be described here in detail.

[0325] The embodiments of the present application further provide a computer program product, which comprises computer program code, when the computer program code is run on a computer, so that the computer executes the method in various possible embodiments as above.

[0326] The embodiment of the present application further provides a chip comprising a memory and a processor, the memory is used for storing a computer program, and the processor is used for calling and running the computer program from the memory, so that a device installed with the chip executes the method in the various possible implementation manners.

[0327] It can be understood that the above scenarios are only examples and do not constitute a limitation on the application scenarios of the technical solutions provided by the embodiments of the present application. The technical solutions provided by the embodiments of the present application are also applicable to other scenarios. For example, those skilled in the art can know that, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0328] The above sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0329] The steps in the method of the embodiments of the present application can be adjusted in sequence, combined and deleted according to actual needs.

[0330] The units in the device of the embodiments of the present application can be combined, divided and deleted according to actual needs.

[0331] In the present application, for the same or similar term concept, technical solution and / or application scenario description, generally only the first time is described in detail, and for the sake of brevity, the repeated description is generally not repeated, and for the understanding of the technical solutions of the present application, the same or similar term concept, technical solution and / or application scenario description which is not described in detail can be referred to the related description before.

[0332] In the present application, the description of each embodiment has its own emphasis, and the part not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0333] The technical features of the technical solutions of the present application can be combined arbitrarily, and in order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the range recorded in the present application.

[0334] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, a controlled terminal, or a network device) to execute the method of each embodiment of the present application.

[0335] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part can be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a storage medium or transferred from one storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available media sets. The available media can be magnetic media (such as floppy disk, storage disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk (SSD)) and the like.

[0336] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: Including steps: S10, using a neural network including at least one partial convolution module to perform filtering processing on at least one image block to obtain a target image block.

2. The image processing method according to claim 1, wherein: Step S10 includes the following steps: S11, inputting at least one image block, at least one of a quantization parameter and a boundary strength into a neural network comprising at least one partial convolution module; S12, performing convolution processing on at least one image block according to the first convolution layer of the neural network to determine or obtain at least one image block feature to be input into the partial convolution module; S13, performing convolution processing on at least one image block feature according to at least one partial convolution module to determine or obtain a target fusion feature; S14, determining or obtaining at least one target image block after filtering according to the target fusion feature.

3. The image processing method according to claim 2, wherein: Step S13 includes: Inputting at least one image patch feature into at least one partial convolution module; Dividing at least one image block feature by channel or region to determine or obtain at least two first sub-feature sets of the image block; Performing convolution processing on at least one first sub-feature set to determine or obtain a second sub-feature set; The target fusion feature is determined or obtained based on the second sub-feature set and the first sub-feature set that has not been convolved.

4. The image processing method according to claim 3, wherein: The dividing of at least one image block feature by channel or region includes at least one of the following: Dividing at least one image block feature into channels or regions according to parameters of a partial convolution module; Dividing the at least one image block feature by channels according to the number of channels corresponding to at least one convolutional layer in at least one hidden layer of the neural network; An image block segmentation mark is determined or generated according to the image block feature, and at least one image block feature is divided into regions according to the image block segmentation mark.

5. The image processing method according to claim 2, wherein: The image block features include chrominance component image block features and luminance component image block features; And / or, step S13 includes: Performing convolution processing on the chrominance component image block features according to the partial convolution module corresponding to the chrominance component to determine or obtain the fusion features of the chrominance component; Performing convolution processing on the luminance component image block features according to the partial convolution module corresponding to the luminance component to determine or obtain the fusion features of the luminance component; The target fusion feature is determined or obtained based on the fusion feature of the luminance component and the fusion feature of the chrominance component.

6. The image processing method according to claim 2, wherein: Step S14 includes: Perform convolution processing on the target fusion features based on the last convolution layer of the neural network; At least one target image block after filtering is determined or obtained based on the target fusion features after convolution processing and at least one image block input to the neural network.

7. The image processing method according to claim 1, wherein: Also include at least one of the following: Use partial convolution module to perform convolution processing on image block features on some channels; The partial convolution module is used to perform convolution processing on partial areas of image block features on partial channels; The image block is a reconstruction of the block; The partial convolution module includes at least one branch having an identity mapping function; The partial convolution module includes at least one branch of a convolution layer.

8. The image processing method according to claim 1, wherein: Also include at least one of the following: Determine the number of partial convolution modules in the neural network based on the flag bit; Determine the number of branches in some convolution modules in the neural network according to the flag bit; The multiple convolutional layers in the partial convolutional module of the neural network in the training phase are fused to determine or obtain the partial convolutional module in the inference phase.

9. A processing device, characterized in that: include: A memory and a processor, wherein an image processing program is stored in the memory, and when the image processing program is executed by the processor, the steps of the image processing method according to claim 1 are implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, which implements the steps of the image processing method according to claim 1 when executed by a processor.

Images