A neural network-based image processing method and device

Abstract

The application discloses a neural network-based image processing method and device, relates to the technical field of image processing, and is used for reducing image processing delay on the basis of ensuring image quality. The method comprises the following steps: inputting multiple frames of to-be-processed images into a first neural network for operation, so as to obtain a first image; inputting multiple image groups into multiple second neural networks for operation, so as to respectively obtain multiple frames of second images, wherein each image group comprises the first image and one frame of image in the multiple frames of to-be-processed images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method and apparatus based on neural networks. Background Technology

[0002] With the development of science and technology, mobile terminals such as smartphones and tablets with photo and video recording functions have been widely used. During the photo or video recording process, mobile terminals perform image signal processing (ISP) on the image signals.

[0003] The primary function of an Image Signal Processor (ISP) is to perform post-processing on the image signals output from the front-end image sensor. Thanks to the ISP, images obtained under different optical conditions can better reproduce scene details. The ISP processing flow is as follows: Figure 1 As shown, the natural scene 101 obtains a Bayer image through the lens 102, then obtains an analog electrical signal 105 through the sensor 103 and photoelectric conversion 104, and further obtains a digital electrical signal (i.e., the raw image) 107 through noise reduction and analog-to-digital conversion (A / D) 106, which then enters the digital signal processing chip 100. The steps in the digital signal processing chip 100 are the core steps of ISP processing. The digital signal processing chip 100 generally includes black level compensation (BLC) 108, lens shading correction 109, bad pixel correction (BPC) 110, demosaicing 111, Bayer domain noise reduction 112, auto white balance (AWB) 113, ygamma 114, auto exposure (AE) 115, and auto focus (AF). Figure 1 (Not shown in the image) 116, Color Correction (CC) 117, Gamma Correction 118, Color Gamut Conversion 119, Color Denoising / Detail Enhancement 110, Color Enhancement (CE) 121, Formatter 122, Input / Output (I / O) Control, etc.

[0004] Currently, deep learning is being applied more and more widely, and deep learning-based image ISPs have achieved certain results in many tasks. Deep learning-based ISPs process image data through neural networks before outputting the result. However, the processing complexity of neural networks is generally very high. While they can achieve the desired results in non-real-time processing scenarios, they typically present problems such as energy consumption and runtime in scenarios requiring real-time processing.

[0005] Therefore, neural network-based ISPs need further optimization. Summary of the Invention

[0006] This application provides an image processing method and apparatus based on neural networks, aiming to optimize the performance of image signal processing based on neural networks.

[0007] In a first aspect, a neural network-based image processing method is provided, which uses a first neural network and a second neural network to process multiple frames of images to be processed and outputs a second image. The steps of the method are as follows: inputting multiple frames of images to be processed into the first neural network for operation to obtain a first image; inputting multiple image groups into multiple second neural networks for operation to obtain multiple frames of second images respectively, wherein each image group includes the first image and one frame of the multiple frames of images to be processed.

[0008] The neural network-based image processing method provided in this application obtains a first image by processing multiple frames of images to be processed through a first neural network, thus acquiring the common image features of the multiple frames. The first image and one frame of images to be processed are then processed through a second neural network to obtain a second image, thereby obtaining multiple second images. Because the first and second neural networks are used separately to process the multiple frames of images, and the first image is used in the processing of the second neural network, the computational complexity of the second neural network is reduced, while ensuring the quality of image processing.

[0009] In one possible implementation, multiple image groups are input into multiple second neural networks for processing, including: inputting one frame of the multiple images to be processed into the second neural network for processing to obtain a third image; and merging the first image and the third image to obtain a second image.

[0010] Optionally, the first neural network includes an image output layer and multiple feature map output layers. The image output layer outputs a first image, and the feature map output layers output multiple intermediate feature maps. These intermediate feature maps are used to participate in the operation of the second neural network to obtain a third image.

[0011] Optionally, the complexity of the second neural network is lower than that of the first neural network.

[0012] Optionally, the multi-frame images to be processed include multiple temporally adjacent images.

[0013] Optionally, the first neural network has a greater ability to process static areas of the image than the second neural network.

[0014] In one possible design, the first neural network is used to process the stationary regions of the image to be processed.

[0015] In one possible design, a second neural network is used to process the motion regions of the image to be processed.

[0016] In one possible design, a first neural network and a second neural network constitute an image processing system, which is used to denoise and eliminate mosaic effects from the image to be processed.

[0017] In a second aspect, a neural network-based image processing apparatus is provided. This apparatus can be a mobile terminal, a device within the mobile terminal (e.g., a chip, chip system, or circuit), or a device compatible with the mobile terminal. In one design, the apparatus may include modules corresponding to the methods / operations / steps / actions described in the first aspect. These modules can be hardware circuits, software, or a combination of hardware circuits and software. The apparatus processes multiple frames of images to be processed to obtain a second image. In one design, the apparatus may include a computation module. Exemplarily: the computation module is used to input multiple frames of images to be processed into a first neural network for computation to obtain a first image; the computation module is also used to input multiple image groups into multiple second neural networks for computation to obtain multiple frames of second images, wherein each image group includes the first image and one frame from the multiple frames of images to be processed.

[0018] In one possible implementation, the computation module is used to: input one frame of the multi-frame images to be processed into a second neural network for computation to obtain a third image; and merge the first image and the third image to obtain a second image.

[0019] Optionally, the first neural network includes an image output layer and multiple feature map output layers. The image output layer outputs a first image, and the feature map output layers output multiple intermediate feature maps. These intermediate feature maps are used to participate in the operation of the second neural network to obtain a third image.

[0020] Optionally, the complexity of the second neural network is lower than that of the first neural network.

[0021] Optionally, the multi-frame images to be processed include multiple temporally adjacent images.

[0022] Optionally, the first neural network has a greater ability to process static areas of the image than the second neural network.

[0023] In one possible design, the first neural network is used to process the stationary regions of the image to be processed.

[0024] In one possible design, a second neural network is used to process the motion regions of the image to be processed.

[0025] In one possible design, a first neural network and a second neural network constitute an image processing system, which is used to denoise and eliminate mosaic effects from the image to be processed.

[0026] The beneficial effects of the second aspect can be referenced from the corresponding effects of the first aspect, and will not be elaborated here.

[0027] Thirdly, embodiments of this application provide an image processing apparatus based on a neural network. The apparatus includes a processor that invokes a set of programs, instructions, or data to execute the method described in the first aspect or any possible design of the first aspect. The apparatus may further include a memory for storing the programs, instructions, or data invoked by the processor. The memory is coupled to the processor, and when the processor executes the instructions or data stored in the memory, it can implement the method described in the first aspect or any possible design.

[0028] Fourthly, embodiments of this application provide a chip system including a processor and potentially a memory, for implementing the methods described in the first aspect or any possible design within the first aspect. This chip system may be composed of chips or may include chips and other discrete devices.

[0029] Fifthly, this application also provides a computer-readable storage medium storing computer-readable instructions that, when executed on a computer, cause the method described in the first aspect or any possible design of the first aspect to be performed.

[0030] In a sixth aspect, this application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in the first aspect or any possible design of the first aspect. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the ISP processing flow in the existing technology;

[0032] Figure 2 This is a schematic diagram of the system architecture provided in one embodiment of the present application;

[0033] Figure 3A schematic diagram illustrating the principle of a neural network provided in one embodiment of this application;

[0034] Figure 4 A flowchart of a neural network-based image processing method provided in an embodiment of this application;

[0035] Figure 5 A schematic diagram illustrating an implementation of image processing according to an embodiment of this application;

[0036] Figure 6 A schematic diagram illustrating an implementation of image processing according to an embodiment of this application;

[0037] Figure 7 A schematic diagram illustrating an implementation of image processing according to an embodiment of this application;

[0038] Figure 8 This is a schematic diagram of an RGrGbB image processing procedure provided in an embodiment of this application;

[0039] Figure 9a This is one of the schematic diagrams of the structure of the first neural network provided in the embodiments of this application;

[0040] Figure 9b This is a second schematic diagram of the structure of the first neural network provided in the embodiments of this application;

[0041] Figure 10a This is one of the schematic diagrams of the structure of the second neural network provided in the embodiments of this application;

[0042] Figure 10b This is one of the schematic diagrams of the structure of the second neural network provided in the embodiments of this application;

[0043] Figure 11a This is a schematic diagram of the structure of a first neural network and a second neural network provided in an embodiment of this application;

[0044] Figure 11b This is a schematic diagram of the structure of a first neural network and a second neural network provided in an embodiment of this application;

[0045] Figure 12 This is a schematic diagram of the structure of a neural network-based image processing device provided in an embodiment of this application;

[0046] Figure 13 This is a schematic diagram of the structure of a neural network-based image processing device provided in an embodiment of this application. Detailed Implementation

[0047] The terms "first," "second," and "third," etc., used in this application specification, claims, and the aforementioned drawings are used to distinguish different objects, not to limit a specific order.

[0048] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0049] The image processing method and apparatus based on neural networks (NN) provided in this application can be applied to electronic devices. These electronic devices can be mobile devices such as mobile terminals, mobile stations (MS), and user equipment (UE), or fixed devices such as landline telephones, desktop computers, or video surveillance systems. The electronic device is an image acquisition and processing device with image signal acquisition and processing capabilities, and it also has ISP processing capabilities. The electronic device can also selectively have wireless connectivity capabilities, providing users with voice and / or data connectivity via handheld devices or other processing devices connected to a wireless modem. For example, the electronic device can be a mobile phone (or "cellular" phone), a computer with a mobile terminal, or a portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile device. It can also be a wearable device (such as a smartwatch, smart bracelet, etc.), a tablet computer, a personal computer (PC), a personal digital assistant (PDA), a point-of-sale (POS), etc. In this application embodiment, the following description uses a mobile terminal as an example of an electronic device.

[0050] Figure 2 The diagram shown is an optional hardware structure diagram of the mobile terminal 200 involved in an embodiment of this application.

[0051] like Figure 2 As shown, the mobile terminal 200 mainly includes a chipset and peripheral devices, among which, Figure 2The components within the solid-lined box—power management unit (PMU), voice codec, short-range module, radio frequency (RF), processing unit, random-access memory (RAM), input / output (I / O), display interface, image signal processor (ISP), sensor hub, and baseband communication module—comprise a chip or chipset. Components such as USB interface, memory, display, battery / mains power, headphones / speakers, antenna, and sensors can be considered peripheral devices. The processing unit, RAM, I / O, display interface, ISP, sensor hub, and baseband components within a chipset can form a system-on-a-chip (SOC), which is the main part of the chipset. All components within a SOC can be integrated into a single complete chip, or some components can be integrated while others are not. For example, the baseband communication module within a SOC can be independent and not integrated with other components. The components within a SOC can be interconnected via buses or other connecting lines. External components of a System-on-a-Chip (SoC), such as the PMU, voice codec, and RF, typically include analog circuitry and are therefore often located outside the SoC and not integrated with each other.

[0052] Figure 2In this module, the PMU (Power Management Unit) is used to connect to external AC power or a battery to power the SOC (System-on-a-Chip), and can use AC power to charge the battery. The voice codec, acting as a voice encoding / decoding unit, connects to headphones or speakers to convert natural analog voice signals to digital voice signals that the SOC can process. Short-range modules may include Wireless Fidelity (WiFi) and Bluetooth, and may optionally include infrared, near-field communication (NFC), FM radio, or GPS modules. The RF module connects to the baseband communication module in the SOC to perform the conversion between air interface RF signals and baseband signals, i.e., frequency mixing. For mobile phones, receiving involves down-conversion, while transmitting involves up-conversion. Both the short-range module and the RF module can have one or more antennas for signal transmission or reception. The baseband is used for baseband communication, including one or more communication modes. It processes wireless communication protocols, including physical layer (layer 1), medium access control (MAC) (layer 2), and radio resource control (RRC) (layer 3), supporting various cellular communication standards such as Long Term Evolution (LTE) and 5G New Radio (NR). The sensor interface is the interface between the SOC and external sensors, used to collect and process data from at least one external sensor, such as an accelerometer, gyroscope, control sensor, or image sensor. The processing unit can be a general-purpose processor, such as a central processing unit (CPU), or one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), microprocessors, or one or more field-programmable gate arrays (FPGAs). The processing unit may include one or more cores and can selectively schedule other units. RAM stores intermediate data during computation or processing, such as intermediate computation data from the CPU and baseband. ISP is used to process data acquired by the image sensor. I / O is used for the SoC to interact with various external interfaces, such as the Universal Serial Bus (USB) interface used for data transmission. Memory can be a single chip or a set of chips.The display screen can be a touch screen, connected to a bus via a display interface. The display interface can be used for data processing before image display, such as overlaying multiple layers to be displayed, caching display data, or controlling and adjusting screen brightness.

[0053] The mobile terminal 200 involved in this embodiment includes an image sensor. This image sensor can collect external signals such as light from the outside world, process these signals, and convert them into sensor signals, i.e., electrical signals. These sensor signals can be static image signals or dynamic video image signals. For example, the image sensor can be a camera.

[0054] The mobile terminal 200 involved in this embodiment of the application also includes an image signal processor. The image sensor collects sensor signals and transmits them to the image signal processor. The image signal processor obtains the sensor signals and can perform image signal processing on the sensor signals to obtain image signals that conform to the characteristics of the human eye in terms of clarity, color, brightness and other aspects.

[0055] It is understood that the image signal processor involved in the embodiments of this application can be a single chip or a group of chips, that is, it can be integrated or independent. For example, the image signal processor included in the mobile terminal 200 can be an integrated ISP chip integrated into the computing processor.

[0056] The mobile terminal 200 involved in this application embodiment has the function of taking photos or recording videos.

[0057] The neural network-based image processing method provided in this application mainly describes how to perform image signal processing based on neural networks.

[0058] To better understand the solutions of the embodiments of this application, the concepts and terms involved in the embodiments of this application will be explained first.

[0059] (1) Neural Network

[0060] In this embodiment, a neural network is used to process multiple frames of images. A neural network is a network structure that mimics the behavioral characteristics of animal neural networks for information processing; it is also simply referred to as a neural network.

[0061] In this context, a neural network can be composed of neural units, where a neural unit can refer to a unit represented by x. s The arithmetic unit takes the intercept 1 as input, and its output can be as shown in formula (1):

[0062]

[0063] Where s = 1, 2, ..., n, n is a natural number greater than 1, Ws For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together; that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0064] like Figure 3 The diagram shows a schematic of a neural network. This neural network 300 has N processing layers, where N ≥ 3 and N is a natural number. The first layer is the input layer 301, responsible for receiving input signals. The last layer is the output layer 303, outputting the processing result. The remaining layers (excluding the first and last layers) are intermediate layers 304. These intermediate layers together form the hidden layer 302. Each intermediate layer in the hidden layer can both receive and output signals, and the hidden layer is responsible for processing the input signals. Each layer represents a logical level of signal processing; through multiple layers, data signals can undergo multi-level logical processing.

[0065] In some feasible embodiments, the input signal of the neural network can be various forms of signals such as voice signals, text signals, image signals, and temperature signals. In this embodiment, the image signal being processed can be various sensor signals such as landscape signals captured by a camera (image sensor), image signals of the community environment captured by a monitoring device, and facial signals of a person obtained by an access control system. The input signal of the neural network includes various other computer-processable engineering signals, which will not be listed here. If deep learning is performed on the image signal using a neural network, the image quality can be improved.

[0066] (2) Deep Neural Networks

[0067] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0068] Although DNNs appear complex, the operation of each layer is actually quite simple, essentially a linear relationship expressed as: y = α(Wx + b), where x is the input vector, y is the output vector, b is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer simply processes the input vector x to obtain the output vector y. Due to the large number of layers in a DNN, the number of coefficients W and offset vectors b is also relatively large. These parameters are defined in DNNs as follows: Taking the coefficient W as an example: Suppose in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... Here, the superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0069] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0070] It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0071] (3) Convolutional Neural Network

[0072] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is that it reduces the connections between layers in the CNN, while also reducing the risk of overfitting.

[0073] The neural network in this application embodiment can be a convolutional neural network, or other types of neural networks, such as a recurrent neural network (RNN).

[0074] It should be understood that the images in the embodiments of this application can be static images (or static pictures) or moving images (or moving pictures). For example, the images in this application can be videos or animated pictures, or they can be static pictures or photographs. For ease of description, in the following embodiments, static images or moving images will be uniformly referred to as images.

[0075] The following describes the neural network-based image processing method provided in the embodiments of this application. This method is executed by a neural network-based image processing device. This neural network-based image processing device can be any device or apparatus with image processing capabilities; for example, this method is executed by… Figure 2 The action may be performed by the mobile terminal 200 shown, or by a device associated with the mobile terminal, or by a portion of the device included in the mobile terminal.

[0076] In this embodiment, multiple neural networks are used for image processing. For example, two neural networks are used to process the image to be processed, referred to as the first neural network and the second neural network. The first neural network and the second neural network conform to the above description of neural networks.

[0077] like Figure 4 As shown in the embodiments of this application, the image processing method based on neural networks includes the following steps.

[0078] S401. Input multiple frames of images to be processed into the first neural network for processing to obtain the first image.

[0079] S402. Input multiple image groups into multiple second neural networks for processing to obtain multiple frames of second images. Each image group includes a first image and one frame of the multiple frames to be processed.

[0080] like Figure 5As shown, taking n frames of images to be processed as an example, where n is an integer greater than or equal to 2, the process involves inputting the n frames of images to be processed into a first neural network to obtain a first image. The first image and the first frame of images to be processed are then input into a first second neural network, the first image and the second frame of images to be processed are input into a second second neural network, and so on, until the first image and the nth frame of images to be processed are input into the nth second neural network. Each second neural network receives the first image and one frame of images to be processed, and each second neural network outputs one frame of images to be processed; that is, the first second neural network outputs the first frame of images to be processed, the second second neural network outputs the second frame of images to be processed, and so on, until the nth second neural network outputs the nth frame of images to be processed.

[0081] pass Figure 4 The method shown involves processing multiple frames of images to be processed using a first neural network to obtain a first image, which captures the common image features of the multiple frames. The first image and one frame of images to be processed are then processed using a second neural network to obtain a second image, thus generating multiple second images. Because the first and second neural networks are used separately to process the multiple frames of images, and the first image is incorporated into the processing of the second neural network, the computational complexity of the second neural network is reduced, while maintaining image processing quality.

[0082] For example, the first neural network is used to process still regions of multiple frames of images. The first image can be an image of a still region shared by multiple frames. Since the features of the still regions account for a high proportion of the network complexity, the first neural network processes the features of the still regions first, and the processed results are used as intermediate results to input into the second neural network, thus reducing the complexity requirement of the second neural network. By using the two neural networks in combination, a lower complexity can be achieved when processing multiple frames of images compared to using a single neural network.

[0083] Optionally, a second neural network is used to process the motion regions of multiple frames of images to be processed.

[0084] The following describes some optional designs of the neural network-based image processing method provided in the embodiments of this application.

[0085] In one possible implementation, such as Figure 6As shown, n frames of images to be processed are input into a first neural network to obtain a first image. The first image and the first frame of images to be processed are input into a first second neural network, the first image and the second frame of images to be processed are input into a second second neural network, and so on, until the first image and the nth frame of images to be processed are input into the nth second neural network. Understandably, each second neural network receives the first image and one frame of images to be processed. Further, each second neural network outputs a third frame of images. The first image and the third image are then merged to obtain a second image. Each second neural network outputs a second frame of images; that is, the first second neural network outputs the first frame of the second image, the second second neural network outputs the second frame of the second image, and so on, until the nth second neural network outputs the nth frame of the second image.

[0086] Optionally, merging the first and third images can also be considered as combining the first and third images, such as performing matrix addition on the first and third images to obtain the second image. For example, suppose the first image is an image of the static region after processing by the first neural network, and the third image is an image of the moving region after processing by the second neural network. Then, merging the first and third images means merging the processed static region image and the processed moving region image to obtain the complete second image.

[0087] In one possible implementation, the first neural network and / or the second neural network do not divide or identify the image to be processed into static and / or moving regions, but rather process the image as a whole. For example, the characteristics of the first neural network itself cause it to process image regions with static region characteristics in the image to be processed, while the characteristics of the second neural network itself cause it to process image regions with moving region characteristics in the image to be processed and the intermediate images processed by the first neural network. For another example, the characteristics of the first neural network itself cause it to process image regions with static region characteristics with higher intensity and image regions with moving region characteristics with lower intensity, while the characteristics of the second neural network itself cause it to process image regions with moving region characteristics with higher intensity and image regions with static region characteristics in the image to be processed and the intermediate images processed by the first neural network with lower intensity. Correspondingly, the first image can be the image of a static region processed by the first neural network, and the third image can be the image of a moving region processed by the second neural network. It should be understood that, for ease of description, the above implementation is also briefly described as the first neural network processing the static region image and the second neural network processing the moving region image.

[0088] In another possible implementation, such as Figure 7 As shown above, Figure 6 The difference in the implementation method is that the second neural network also receives intermediate feature maps output by the first neural network. These intermediate feature maps are used in the computation of the second neural network to obtain the third image.

[0089] For example, a frame of the image to be processed and an intermediate feature map can be concatenated or added to obtain a matrix of images to be processed. This matrix is ​​then input into a second neural network for processing to obtain a third image. It can be understood that concatenating a frame of the image to be processed and an intermediate feature map can be seen as a processing step within the second neural network. The input to the second neural network is a single matrix, namely the matrix of images to be processed.

[0090] Optionally, the second neural network also receives multiple intermediate feature maps output by the first neural network. A single frame of the image to be processed and the intermediate feature map of the first frame can be concatenated or added to obtain a matrix of images to be processed. This matrix is ​​then input into the second neural network for computation to obtain intermediate feature maps of the second neural network. These intermediate feature maps are then concatenated or added to the intermediate feature maps of the first neural network, and further processed by the remaining layers of the second neural network to obtain the third image.

[0091] Optionally, the first neural network includes an image output layer and multiple feature map output layers. The image output layer outputs a first image. The feature map output layers output multiple intermediate feature maps.

[0092] In this embodiment, the first image, the second image, and the third image are not specifically distinguished. For example, the color, texture, and other features of the first image, the second image, and the third image are not specifically distinguished.

[0093] In this embodiment, the multiple frames of images to be processed include multiple frames of temporally adjacent images. Optionally, the multiple frames of temporally adjacent images include multiple frames of temporally consecutive images. After the multiple frames of images to be processed are processed by multiple second neural networks, the processed images are also corresponding multiple frames. For example, the second images obtained after processing by multiple second neural networks are multiple frames, with each frame of the image to be processed corresponding to one frame of the second image. Each frame of the second image corresponds to a first image and a third image.

[0094] In this embodiment, the format of the image to be processed can optionally be RGB format, YUV format, or Bayer format. No limitation is made in this application.

[0095] For example, if there are four frames of images to be processed, these four frames are input into a first neural network for processing to obtain a first image. The four frames correspond to the first image. Four image groups are then input into four second neural networks for processing to obtain four second images, each group including the first image and one frame from the four frames to be processed. For example, the first frame corresponds to the first second image; the second frame corresponds to the second second image; the third frame corresponds to the third second image; and the fourth frame corresponds to the fourth second image.

[0096] Optionally, the first frame of the image to be processed is input into the first second neural network for processing to obtain the first frame third image. The first image and the first frame third image are then merged to obtain the first frame second image. The second frame of the image to be processed is input into the second second neural network for processing to obtain the second frame third image. The first image and the second frame third image are then merged to obtain the second frame second image. The third frame of the image to be processed is input into the third second neural network for processing to obtain the third frame third image. The first image and the third frame third image are then merged to obtain the third frame second image. The fourth frame of the image to be processed is input into the fourth second neural network for processing to obtain the fourth frame third image. The first image and the fourth frame third image are then merged to obtain the fourth frame second image.

[0097] In this embodiment, the first neural network and the second neural network can be combined to form an image processing system. This image repair system is used to process the image to be processed to improve the quality of the image or video. The processing may include noise reduction and mosaic effect elimination.

[0098] Generally, the complexity of the first neural network is higher than that of the second neural network. For example, the first neural network is more capable of processing static areas of an image than the second neural network.

[0099] In some technologies, multiple frames of images are often synthesized into a single output frame using neural networks to improve image or video quality. However, such neural networks require high complexity, and in video scenarios, high processing speeds are necessary. For example, mobile terminal video processing requires a processing speed of 30 frames per second (fps) for 8K resolution video. Given the high speed requirements of video scenarios, using neural networks to synthesize multiple frames into a single output frame presents significant challenges in terms of computational complexity and resource consumption, as well as substantial latency. Conversely, simply reducing the complexity of the neural network to use less complex networks can compromise image or video quality.

[0100] In this embodiment, a first neural network is used to handle the complex computational burden between multiple image frames, while a second neural network is used to handle the lower computational burden of each individual image frame within the multi-frame image set. The processed multi-frame image is then output. This distributes the combined computational power of the first and second neural networks across the multiple image frames, reducing the processing complexity of each frame compared to the previous schemes while maintaining image or video quality. For example, if the first image represents a static region and the third image represents a moving region, the first neural network processes the static regions of the multi-frame image set, while the second neural network processes the moving regions. This combined processing by the two neural networks results in a lower complexity image processing system while ensuring image or video quality. This improves the application of deep learning technology in the field of image signal processing.

[0101] The following explanation uses convolutional neural networks as an example, with the first and second neural networks being convolutional neural networks. Assume there are 4 frames of the image to be processed, and 4 frames of the second image. The image to be processed is in Bell format, specifically RGrGbB format, where one frame of an RGrGbB image includes 4 channels (R, Gr, Gb, B). After the 4 frames of the image to be processed pass through the image processing system, 4 frames of processed images are output. The image processing system includes the first neural network and the second neural network.

[0102] like Figure 8 As shown, four consecutive RGrGbB images to be processed are split into 4*4=16 channels. These 16 channels include (R1, Gr1, Gb1, B1, R2, Gr2, Gb2, B2, R3, Gr3, Gb3, B3, R4, Gr4, Gb4, B4). The four consecutive RGrGbB images are input into the first neural network to obtain the first image (4 channels (R, Gr, Gb, B)).

[0103] The first frame RGrGbB image (4 channels (R1, Gr1, Gb1, B1)) is input into the first second neural network to obtain the first frame third image (4 channels (R1', Gr1', Gb1', B1')). The first image (R, Gr, Gb, B) and the first frame third image (4 channels (R1', Gr1', Gb1', B1')) are merged to obtain the first frame second image (4 channels (R1”, Gr1”, Gb1”, B1”)).

[0104] The second frame RGrGbB image (4 channels (R2, Gr2, Gb2, B2)) is input into the second neural network to obtain the second frame third image (4 channels (R2', Gr2', Gb2', B2')). The first image (R, Gr, Gb, B) and the second frame third image (4 channels (R2', Gr2', Gb2', B2')) are merged to obtain the second frame second image (4 channels (R2”, Gr2”, Gb2”, B2”)).

[0105] The 3rd frame RGrGbB image (4 channels (R3, Gr3, Gb3, B3)) is input into the third second neural network to obtain the 3rd frame third image (4 channels (R3', Gr3', Gb3', B3')). The first image (R, Gr, Gb, B) and the 3rd frame third image (4 channels (R3', Gr3', Gb3', B3')) are merged to obtain the 3rd frame second image (4 channels (R3”, Gr3”, Gb3”, B3”)).

[0106] The 4th frame RGrGbB image (4 channels (R4, Gr4, Gb4, B4)) is input into the 4th second neural network to obtain the 4th frame third image (4 channels (R4', Gr4', Gb4', B4')). The first image (R, Gr, Gb, B) and the 4th frame third image (4 channels (R4', Gr4', Gb4', B4')) are merged to obtain the 4th frame second image (4 channels (R4”, Gr4”, Gb4”, B4”)).

[0107] An example, the architecture in the first neural network Figure 9a and Figure 9b As shown, because the attached diagram of the first neural network is too large, the first neural network is split into two parts, each consisting of... Figure 9a and Figure 9b As shown. Figure 9a and Figure 9b Together they form the architecture of the first neural network. Figure 9a After adding, connect Figure 9b The first layer in the middle.

[0108] exist Figure 9a and Figure 9b In the diagram, convolutional layers are represented by rectangles. Within a rectangle, `Conv2d + bias stride = 23x3_16_32` represents a convolutional layer. Here, `Conv2d` represents 2D convolution, `bias` represents the bias term, `1x1 / 3x3` represents the kernel size, `stride` represents the stride, and `_32_16` represents the number of input and output feature maps. `32` indicates that the input feature maps for this layer are 32, and `16` indicates that the output feature maps for this layer are 16.

[0109] `split` means splitting the feature map along the channel dimension. `split 2` means splitting the image along the feature map dimension. For example, 32 input feature maps will be transformed into two images with 16 feature maps each after the above operation.

[0110] concat represents a skip chain layer, indicating that images will be merged along the feature map dimension. For example, two images with 16 feature maps will be merged into one image with 32 feature maps.

[0111] add represents the matrix addition operation.

[0112] Figure 9a and Figure 9b The first neural network shown is a typical convolutional neural network, which can effectively handle static regions in multi-frame images. Assume that the first neural network inputs the image to be processed (image 4) and outputs the first image.

[0113] Optionally, instead of a typical convolutional neural network, a multi-branch neural network can be used. The first convolutional layer of the neural network can also employ group convolution. Group convolution is a special type of convolutional layer. Assume the previous layer's output feature map has N channels (N channels), meaning the previous layer has N convolutional kernels. Further assume the number of groups is M. The operation of this group convolution layer is as follows: first, the N channels are divided into M groups. Each group corresponds to N / M channels. Each group convolution is performed independently, and the output feature maps are then concatenated as vectors to form the output channels of this layer. Using group convolution can achieve similar or identical technical effects to using a branching approach.

[0114] For example, an architecture where four second neural networks process four images to be processed. Figure 10a and Figure 10b As shown. Figure 10a The first image output from the input is given to Figure 10b The first layer in. In Figure 10a and Figure 10b In the diagram, convolutional layers are represented by rectangles. For a detailed explanation, please refer to [reference needed]. Figure 9a and Figure 9b The explanation in the text.

[0115] like Figure 10a As shown, the first frame of the image to be processed is input into the first second neural network to obtain the first frame of the third image. The first image and the first frame of the third image are then merged to obtain the first frame of the second image.

[0116] The second frame of the image to be processed is input into the second neural network to obtain the third frame of the image. The first image and the third frame of the image are then merged to obtain the second frame of the image.

[0117] like Figure 10b As shown, the third frame of the image to be processed is input into the third second neural network to obtain the third frame of the third image. The first image and the third frame of the third image are then merged to obtain the second frame of the third image.

[0118] The fourth frame of the image to be processed is input into the fourth second neural network to obtain the fourth frame of the third image. The first image and the fourth frame of the third image are then merged to obtain the fourth frame of the second image.

[0119] Optional, such as Figure 11a and Figure 11b As shown, the first neural network can also input intermediate feature maps, which are then fed into the second neural network. These intermediate feature maps participate in the computation of the second neural network to obtain the third image. For example, the intermediate feature map output from the second convolutional layer of the first neural network is used to perform vector concatenation with the image output from the first convolutional layer of the second neural network to obtain the processed image. Similarly, the intermediate feature map output from the fourth convolutional layer of the first neural network is used to perform vector concatenation with the intermediate feature map output from the third convolutional layer of the second neural network to obtain the processed image.

[0120] In this embodiment, the first neural network and the second neural network need to be trained before use. During the training process, the training data may include training images and ground truth images.

[0121] When training the model of the first neural network: First, the collected training images are processed using the ground truth image of the first image to obtain and output images. The output images are compared with the ground truth image of the first image until the network converges, thus completing the training of the model of the first neural network. Network convergence can be defined as, for example, the difference between the output image and the ground truth image of the first image being less than a set first threshold.

[0122] The parameters of the first image obtained from training the first neural network are fixed. The collected training image is processed using the ground truth image of the third image to obtain and output an image. The output image is compared with the ground truth image of the third image until the network converges, thus completing the training of the model of the second neural network. Here, network convergence can be defined as the difference between the output image and the ground truth image of the third image being less than a set second threshold.

[0123] In this embodiment, an image processing system is constructed using a first neural network and a second neural network. This system processes multiple frames of images to be processed and outputs multiple frames of processed images. The complexity of the second neural network is lower than that of the first neural network. Compared to some techniques that process multiple frames of images into a single frame using a basic network, the computational load of the image processing system for each frame of the image to be processed is reduced to a certain extent. This reduces image processing latency and ensures image or video quality. The computational power of two neural networks processing multiple frames of images is illustrated below. Assume there are 4 frames of images to be processed, and the processed images output by the first and second neural networks are also 4 frames. After processing by the basic network, one frame is output. The first neural network is as follows... Figure 9a and Figure 9b As shown, the second neural network is as follows Figure 10a and Figure 10b As shown.

[0124] The computational cost of the first neural network is approximately the same as that of the basic network, around 12,000 MAC. For example, the network complexity calculation process for the basic network is as follows:

[0125] (23*32*1*1+32*16*3*3) / 4#1336

[0126] +16*32*3*3 / 16#288

[0127] +(32*32*3*3) / 16#576

[0128] +32*64*3*3 / 64#288

[0129] +(64*96*3*3+(48*48*3*3*2+96*96*1*1*1)*2+96*64*3*3+32*32*3*3*2+64*64*1*1*1+64*64*3*3*1) / 64#4240

[0130] +(64*32*2*2) / 16+(concat)#512

[0131] +(64*32*3*3) / 16#1152

[0132] +(32*16*2*2) / 4+(concat)#512

[0133] +(32*16*3*3) / 4#1152

[0134] +(16*16*3*3+16*16*3*3+16*4*3*3) / 4#1296

[0135] =11352

[0136] The calculation process for the network complexity of the second neural network is given below:

[0137] (4*16*3*3) / 4#144

[0138] +16*32*3*3 / 16#288

[0139] +(32*32*3*3) / 16#576

[0140] +32*64*3*3 / 64#288

[0141] +(64*64*3*3) / 64#576

[0142] +(64*32*2*2) / 16+(concat)#512

[0143] +(64*32*3*3) / 16#1152

[0144] +(32*16*2*2) / 4+(concat)#512

[0145] +(32*4*3*3) / 4#288

[0146] =4336

[0147] It can be seen that the computational cost of the second neural network is around 4000, let's assume it's 4000.

[0148] Therefore, when 4 frames of images to be processed are input and 4 frames of processed images are output simultaneously, the computational load of the image processing system is (4000*4+12000) / 4 = 7000; when 8 frames of images to be processed are input and 8 frames of processed images are output simultaneously, the computational load of the image processing system is (4000*8+12000) / 8 = 5500; when 16 frames of images to be processed are input and 16 frames of processed images are output simultaneously, the computational load of the image processing system is (4000*16+12000) / 16 = 4700. All of these are less than the computational load of 12000 required to process multiple frames into one frame using a basic network. It can be seen that the multi-frame input and multi-frame output scheme using the first and second neural networks provided in this application can reduce the computational load, thereby reducing the latency of image processing, and can meet the latency requirements of video image processing in video scenarios. The network computing power requirement for a video with a resolution of 8,000 (K) pixels and a frame rate of 30 frames per second is approximately 50,000 MAC. In this embodiment of the application, when outputting 8 frames, the computational load specified by the image processing system can basically meet the network computing power requirement of 8K 30 video.

[0149] It should be noted that the examples in the various application scenarios of this application only illustrate some possible implementation methods, and are intended to provide a better understanding and explanation of the methods in this application. Those skilled in the art can derive examples of some evolved forms based on the indication methods of the reference signals provided in the application.

[0150] To achieve the functions of the methods provided in the embodiments of this application, a neural network-based image processing device may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0151] like Figure 12 As shown, based on the same technical concept, this application embodiment also provides a neural network-based image processing device 1200, which can be a mobile terminal or any device with image processing capabilities. In one design, the neural network-based image processing device 1200 may include modules corresponding to the methods / operations / steps / actions described in the above method embodiments. These modules may be hardware circuits, software, or a combination of hardware circuits and software. In another design, the neural network-based image processing device 1200 may include a computation module 1201.

[0152] The operation module 1201 is used to input multiple frames of images to be processed into a first neural network for operation to obtain a first image; and to input multiple image groups into multiple second neural networks for operation to obtain multiple frames of second images respectively, wherein each image group includes the first image and one frame of the multiple frames of images to be processed.

[0153] The module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in each embodiment of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0154] Based on the same technological concept, such as Figure 13As shown in the illustration, this application also provides a neural network-based image processing apparatus 1300. The neural network image processing apparatus 1300 includes a processor 1301. The processor 1301 is used to invoke a set of programs to execute the above-described method embodiments. The neural network image processing apparatus 1300 also includes a memory 1302, which stores program instructions and / or data executed by the processor 1301. The memory 1302 and the processor 1301 are coupled. The coupling in this application embodiment is an indirect coupling or communication connection between devices, units, or modules, and can be electrical, mechanical, or other forms, used for information interaction between devices, units, or modules. The processor 1301 may operate collaboratively with the memory 1302. The processor 1301 may execute program instructions stored in the memory 1302. The memory 1302 may be included in the processor 1301.

[0155] The neural network-based image processing device 1300 can be a chip system. In this embodiment, the chip system can be composed of chips or may include chips and other discrete devices.

[0156] The processor 1301 is used to input multiple frames of images to be processed into a first neural network for processing to obtain a first image; and to input multiple image groups into multiple second neural networks for processing to obtain multiple frames of second images respectively, wherein each image group includes the first image and one frame of the multiple frames of images to be processed.

[0157] The processor 1301 can be a general-purpose processor, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0158] The memory 1302 can be non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or it can be volatile memory, such as random-access memory (RAM). Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. The memory in the embodiments of this application can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0159] Some or all of the operations and functions described in the above method embodiments of this application can be implemented using chips or integrated circuits.

[0160] This application also provides a chip, including a processor, for supporting the neural network-based image processing device in implementing the functions involved in the above-described method embodiments. In one possible design, the chip is connected to a memory or the chip includes a memory for storing necessary program instructions and data for the communication device.

[0161] This application provides a computer-readable storage medium storing a computer program, the computer program including instructions for performing the above-described method embodiments.

[0162] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the above-described method embodiments.

[0163] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0167] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0168] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A neural network-based image processing method, characterized by, The method comprises: inputting a plurality of frames of to-be-processed images into a first neural network for operation to obtain a first image; inputting a plurality of image groups into a plurality of second neural networks for operation to obtain a plurality of frames of second images respectively, wherein each of the image groups comprises the first image and one frame of the plurality of frames of to-be-processed images; the operation of inputting the plurality of image groups into the plurality of second neural networks comprises: inputting one frame of the plurality of frames of to-be-processed images into the second neural network for operation to obtain a third image; merging the first image and the third image to obtain the second image.

2. The method of claim 1, wherein, The first neural network comprises one image output layer and a plurality of feature map output layers, the image output layer outputs the first image, and the feature map output layers output a plurality of intermediate feature maps, the plurality of intermediate feature maps are used for participating in operation of the second neural network to obtain a third image.

3. The method according to claim 1 or 2, characterized in that, The plurality of frames of to-be-processed images comprise a plurality of frames of time-domain adjacent images.

4. The method according to any one of claims 1-3, characterized in that, The first neural network is used for processing a static region of the to-be-processed image.

5. The method of claim 1, wherein, The second neural network is used for processing a motion region of the to-be-processed image.

6. The method of claim 1, wherein, The processing capability of the first neural network for the static region of the image is greater than that of the second neural network.

7. The method of claim 1, wherein, The first neural network and the second neural network constitute an image processing system, and the image processing system is used for denoising and eliminating a mosaic effect of the to-be-processed image.

8. A neural network-based image processing apparatus, characterized by comprising: The method comprises: an operation module, configured to input a plurality of frames of to-be-processed images into a first neural network for operation to obtain a first image; the operation module is further configured to input a plurality of image groups into a plurality of second neural networks for operation to obtain a plurality of frames of second images respectively, wherein each of the image groups comprises the first image and one frame of the plurality of frames of to-be-processed images; the operation module is configured to: input one frame of the plurality of frames of to-be-processed images into the second neural network for operation to obtain a third image; merge the first image and the third image to obtain the second image.

9. The apparatus of claim 8, wherein, The first neural network comprises one image output layer and a plurality of feature map output layers, the image output layer outputs the first image, and the feature map output layers output a plurality of intermediate feature maps, the plurality of intermediate feature maps are used for participating in operation of the second neural network to obtain a third image.

10. The apparatus of any one of claims 8-9, wherein, The plurality of frames of to-be-processed images comprise a plurality of frames of time-domain adjacent images.

11. The apparatus of claim 8, wherein, The first neural network is used for processing a static region of the to-be-processed image.

12. The apparatus of claim 8, wherein, The second neural network is used for processing a motion region of the to-be-processed image.

13. The apparatus of claim 8, wherein, The processing capability of the first neural network for the static region of the image is greater than that of the second neural network.

14. The apparatus of claim 8, wherein, The first neural network and the second neural network constitute an image processing system, and the image processing system is used for denoising and eliminating a mosaic effect of the to-be-processed image.

15. A chip, characterized by The chip is connected with the memory, and is used for reading and executing a software program stored in the memory, so as to realize the method in any one of claims 1-7.

16. A neural network-based image processing apparatus, characterized by comprising: The chip is connected with the memory, and is used for reading and executing a software program stored in the memory, so as to realize the method in any one of claims 1-7. The chip is connected with the memory, and is used for reading and executing a software program stored in the memory, so as to realize the method in any one of claims 1-7.

17. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions which, when executed on the neural network based image processing device, cause the neural network based image processing device to perform the method of any one of claims 1-7.

Images