Image processing method and device

Abstract

The present application provides an image processing method and device, which relates to the field of computer processing technology, and is used to improve the flexibility and adjustability of image processing, and can improve the clarity and accuracy of the final generated image. The method includes: preprocessing the original RAW data to obtain the first RAW data and the second RAW data, the resolution of the image corresponding to the second RAW data is less than the resolution of the image corresponding to the first RAW data; performing local pixel processing on the first RAW data to obtain the first image data, and performing global pixel processing on the second RAW data to obtain the second image data; generating the target image data according to the first image data and the second image data.

Description

Technical Field

[0001] The present application relates to the field of computer processing technology, and in particular to an image processing method and device. Background Art

[0002] As the visual basis of human perception of the world, images are an important means for humans to obtain, express and transmit information. Image Signal Processor (ISP) is an important component of camera equipment. When we take a photo, the camera can obtain the Bayer image corresponding to the target scene through the lens, and convert the Bayer image from analog to digital to obtain a digital image signal (i.e. RAW data). The RAW data is optimized by a series of calculations through the ISP, such as noise reduction, color adjustment, brightness adjustment, and exposure adjustment, and finally generates the target image displayed on the display.

[0003] At present, image processing is usually implemented by deep learning technology. Deep learning is mainly based on various methods of artificial neural networks to achieve computational optimization, and its application is becoming more and more extensive. In the prior art, multiple processing steps such as noise reduction, color adjustment and brightness adjustment are usually performed in a serial manner. For example, the processing of RAW data to the target image is achieved through a neural network that can implement multiple processing steps (multiple processing steps are performed in a serial manner), or the processing of RAW data to the target image is achieved through multiple neural networks in series (each neural network can implement different processing steps).

[0004] However, in the above-mentioned method of processing images through one neural network or multiple serial neural networks, the relevant parameters of each neural network are fixed, so the effect of the target image is also fixed, and it is impossible to adjust when the effect of the target image is not good or different image effects need to be switched. In addition, in the above-mentioned method of processing images through multiple serial neural networks, the input data of the latter neural network is completely dependent on the output data of the previous neural network, that is, the previous and next neural networks are highly dependent, which increases the difficulty of training the later neural network. Summary of the invention

[0005] The embodiments of the present application provide an image processing method and device for improving the flexibility, adjustability and processing accuracy of image processing, while improving the clarity of the processed image.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, an image processing method is provided, the method comprising: preprocessing original RAW data to obtain first RAW data and second RAW data, the resolution of an image corresponding to the second RAW data being smaller than the resolution of an image corresponding to the first RAW data; performing local pixel processing on the first RAW data to obtain first image data, and performing global pixel processing on the second RAW data to obtain second image data; generating target image data based on the first image data and the second image data.

[0008] In the above technical scheme, the local pixel processing step and the global pixel processing step are independent of each other and executed in parallel, and the parameters of the local pixel processing and the global pixel processing can be adjusted as needed to improve the flexibility and adjustability of image processing; at the same time, the image data of the local pixels and the image data of the global pixel processing are not dependent on each other, so that the original image information of the processed image is retained to the greatest extent, the image processing accuracy is improved, and the clarity of the final generated image is improved.

[0009] In a possible implementation of the first aspect, the first image data includes linear RGB image data, and local pixel processing is performed on the first RAW data to obtain the first image data, including: performing at least one of noise reduction processing or demosaicing processing on local pixels of the first RAW data to obtain the linear RGB image data. In the above possible implementation, performing noise reduction processing or demosaicing processing on local pixels of the raw image processing can improve the clarity of the target image.

[0010] In a possible implementation of the first aspect, the first image data further includes a first brightness ratio matrix, and local pixel processing is performed on the first RAW data to obtain the first image data, further including: brightness processing is performed on local pixels of the first RAW data to obtain the first brightness ratio matrix. In the above possible implementation, brightness processing is performed on the pre-processed image data, and is performed independently and in parallel with other local pixel processing and global pixel processing steps, and parameters of the local pixel processing can be adjusted as needed, thereby improving the flexibility and adjustability of image processing.

[0011] In a possible implementation of the first aspect, the second image data includes a color conversion matrix, and performing global pixel processing on the second RAW data to obtain the second image data includes: performing color processing on the global pixels of the second RAW data to obtain the color conversion matrix. In the above possible implementation, color processing is performed on the pre-processed image data, and other local pixel processing and global pixel processing steps are independent and executed in parallel, and parameters of the global pixel processing can be adjusted as needed, thereby improving the flexibility and adjustability of image processing.

[0012] In a possible implementation of the first aspect, the second image data further includes a second brightness ratio matrix, and performing global pixel processing on the second RAW data to obtain the second image data includes: performing brightness processing on the global pixels of the second RAW data to obtain the second brightness ratio matrix. In the above possible implementation, brightness processing is performed on the pre-processed image data, and is independent and executed in parallel with other local pixel processing and global pixel processing steps, and parameters of the global pixel processing can be adjusted as needed, thereby improving the flexibility and adjustability of image processing.

[0013] In a possible implementation of the first aspect, generating target image data according to the first image data and the second image data includes: generating the target image data according to one of the first brightness ratio matrix and the second brightness ratio matrix, the linear RGB image data and the color conversion matrix. In the above possible implementation, the linear RGB image data, the color conversion matrix, the first brightness ratio matrix or the second brightness ratio matrix are all generated based on the original image data processing, which can retain the original image information to the greatest extent and are not dependent on each other, which can improve the image processing accuracy, thereby improving the clarity of the finally generated image.

[0014] In a possible implementation of the first aspect, preprocessing the raw image data to obtain the first RAW data and the second RAW data includes: performing black level correction, normalization processing and channel splitting processing on the raw image data to obtain the first RAW data; performing downsampling processing, black level correction, normalization processing and channel splitting processing on the raw image data to obtain the second RAW data. In the above possible implementation, different data preprocessing is performed on the raw image data in two ways, which facilitates the subsequent parallel processing of global pixel processing and local pixel processing, improves the flexibility of image processing, and can also improve the clarity of the final generated image.

[0015] In a second aspect, an image processing system is provided, which includes: a preprocessing circuit, used to preprocess original RAW data to obtain first RAW data and second RAW data, the resolution of the image corresponding to the second RAW data is smaller than the resolution of the image corresponding to the first RAW data; a local pixel processing network, used to receive the first RAW data output by the preprocessing circuit, and perform local pixel processing on the first RAW data to obtain first image data; a global pixel processing network, used to receive the second RAW data output by the preprocessing circuit, and perform global pixel processing on the second RAW data to obtain second image data; an image synthesis circuit, used to receive the first image data output by the local pixel processing network and the second image data output by the global pixel processing network, and generate target image data based on the first image data and the second image data.

[0016] In a possible implementation of the second aspect, the first image data is linear RGB image data, and the local pixel processing network includes a primary processing network, which is specifically used to: receive the first RAW data output by the preprocessing circuit, and perform at least one of noise reduction processing or demosaicing processing on local pixels of the first RAW data to obtain linear RGB image data.

[0017] In a possible implementation of the second aspect, the first image data also includes a first brightness ratio matrix, the local pixel processing network also includes a first brightness processing network, and the first brightness processing network is specifically used to: receive the first RAW data output by the preprocessing circuit, and perform brightness processing on local pixels of the first RAW data to obtain a first brightness ratio matrix.

[0018] In a possible implementation of the second aspect, the second image data includes a color conversion matrix, the global pixel processing network includes a color processing network, and the color processing network is specifically used to: receive the second RAW data output by the preprocessing circuit, and perform color processing on the global pixels of the second RAW data to obtain a color conversion matrix.

[0019] In a possible implementation of the second aspect, the second image data also includes a second brightness ratio matrix, the global pixel processing network also includes a second brightness processing network, and the second brightness processing network is specifically used to: receive the second RAW data output by the preprocessing circuit, and perform brightness processing on the global pixels of the second RAW data to obtain a second brightness ratio matrix.

[0020] In a possible implementation of the second aspect, the image synthesis circuit is specifically used to: receive one of a first brightness ratio matrix output by the first brightness processing network and a second brightness ratio matrix output by the second brightness processing network, linear RGB image data output by the primary processing network, and a color conversion matrix output by the color processing network, and generate target image data according to one of the first brightness ratio matrix and the second brightness ratio matrix, the linear RGB image data, and the color conversion matrix.

[0021] In a possible implementation of the second aspect, the preprocessing circuit is specifically used to: perform black level correction, normalization and channel splitting processing on the original image data to obtain first RAW data; downsample, black level correct, normalize and channel splitting processing on the original image data to obtain second RAW data.

[0022] In a third aspect, an image processing device is provided, which includes: a preprocessing unit, used to preprocess original RAW data to obtain first RAW data and second RAW data, the resolution of the image corresponding to the second RAW data is smaller than the resolution of the image corresponding to the first RAW data; a pixel processing unit, used to perform local pixel processing on the first RAW data to obtain first image data, and perform global pixel processing on the second RAW data to obtain second image data; an image synthesis unit, used to generate target image data based on the first image data and the second image data.

[0023] In a possible implementation manner of the third aspect, the first image data is linear RGB image data, and the pixel processing unit is specifically used to: perform at least one of noise reduction processing or demosaicing processing on local pixels of the first RAW data to obtain linear RGB image data.

[0024] In a possible implementation manner of the third aspect, the first image data also includes a first brightness ratio matrix, and the pixel processing unit is further specifically used to: perform brightness processing on local pixels of the first RAW data to obtain the first brightness ratio matrix.

[0025] In a possible implementation manner of the third aspect, the second image data includes a color conversion matrix, and the pixel processing unit is further specifically used to: perform color processing on global pixels of the second RAW data to obtain a color conversion matrix.

[0026] In a possible implementation manner of the third aspect, the second image data also includes a second brightness ratio matrix, and the pixel processing unit is further specifically used to: perform brightness processing on global pixels of the second RAW data to obtain a second brightness ratio matrix.

[0027] In a possible implementation manner of the third aspect, the image synthesis unit is specifically configured to generate target image data according to one of the first brightness ratio matrix and the second brightness ratio matrix, the linear RGB image data, and the color conversion matrix.

[0028] In a possible implementation of the third aspect, the preprocessing unit is specifically used to: perform black level correction, normalization and channel splitting processing on the original image data to obtain first RAW data; downsample, black level correct, normalize and channel splitting processing on the original image data to obtain second RAW data.

[0029] In a fourth aspect, an image processing device is provided, which includes a memory and a processor coupled to the memory, the memory stores instructions and data, and the processor executes the instructions in the memory. When the processor executes the stored instructions, the device executes the image processing method provided by the first aspect or any possible implementation of the first aspect.

[0030] In a fifth aspect, a computer storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the image processing method provided by the first aspect or any possible implementation of the first aspect.

[0031] In a sixth aspect, a computer program product is provided. When the computer program product runs on a computer, the computer executes the image processing method provided by the first aspect or any possible implementation of the first aspect.

[0032] It can be understood that any of the image processing devices, systems, readable storage media and computer program products provided above are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0034] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of the data format of original RAW data provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of a preprocessing process for raw RAW data provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of a first image data processing process provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of a brightness processing process provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the structure of an image processing system provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In the present application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b and c can be single or multiple. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, in the embodiments of the present application, the words "first", "second" and the like do not limit the quantity and execution order.

[0042] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0043] Figure 1 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application, the image processing device can be a mobile phone, a tablet computer, a computer, a laptop computer, a video camera, a camera, a wearable device, a vehicle-mounted device, or a terminal device, etc. For the convenience of description, the above-mentioned devices are collectively referred to as image processing devices in this application. The embodiment of the present application is described by taking the image processing device as a mobile phone as an example, and the mobile phone includes: a memory 101, a processor 102, a sensor component 103, a multimedia component 104, an audio component 105, and a power component 106, etc.

[0044] Combine the following Figure 1 A detailed introduction to the various components of the mobile phone:

[0045] The memory 101 can be used to store data, software programs and modules; it mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function, such as a sound playback function, an image playback function, etc.; the data storage area can store data created according to the use of the mobile phone, such as audio data, image data, phone book, etc. In addition, the mobile phone may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0046] The processor 102 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire device. By running or executing software programs and / or modules stored in the memory 101, and calling data stored in the memory 101, it executes various functions of the mobile phone and processes data, thereby monitoring the mobile phone as a whole. In some feasible embodiments, the processor 102 can be a single processor structure, a multi-processor structure, a single-threaded processor, and a multi-threaded processor; in some feasible embodiments, the processor 102 can include a central processing unit, a general processor, a digital signal processor, a microcontroller or a microprocessor. In addition, the processor 102 can further include other hardware circuits or accelerators, such as application-specific integrated circuits, field programmable gate arrays or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 102 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc.

[0047] The sensor component 103 includes one or more sensors for providing status assessment of various aspects for the mobile phone. Among them, the sensor component 103 may include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for detecting the distance between an external object and the mobile phone, or used in imaging applications, that is, becoming a component of a camera or a camera. In addition, the sensor component 103 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor, and the acceleration / deceleration, orientation, opening / closing state of the mobile phone, the relative positioning of the components, or the temperature change of the mobile phone can be detected by the sensor component 103.

[0048] The multimedia component 104 provides a screen of an output interface between the mobile phone and the user, and the screen can be a touch panel, and when the screen is a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In addition, the multimedia component 104 also includes at least one camera, for example, the multimedia component 104 includes a front camera and / or a rear camera. When the mobile phone is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0049] The audio component 105 can provide an audio interface between the user and the mobile phone. For example, the audio component 105 can include an audio circuit, a speaker, and a microphone. The audio circuit can convert the received audio data into an electrical signal and transmit it to the speaker, which converts it into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit and converted into audio data, and then outputs the audio data to be sent to, for example, another mobile phone, or outputs the audio data to the processor 102 for further processing.

[0050] The power supply component 106 is used to provide power to various components of the mobile phone. The power supply component 106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power in the mobile phone.

[0051] Although not shown, the mobile phone may also include a wireless fidelity (WiFi) module, a Bluetooth module, etc., which will not be described in detail in the present embodiment. Those skilled in the art will understand that Figure 1 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0052] Figure 2 A schematic diagram of a flow chart of an image processing method is provided for an embodiment of the present application. The method can be performed by Figure 1 The image processing device shown is used to perform Figure 2 , the method may include the following steps.

[0053] S201: pre-processing original RAW data to obtain first RAW data and second RAW data, wherein the resolution of an image corresponding to the second RAW data is smaller than the resolution of the image corresponding to the first RAW data.

[0054] The original RAW data may also be referred to as the original image data, which may be the Bayer image data corresponding to the target scene, or the RAW data in the Bayer format obtained after the Bayer image data is converted from analog to digital. Figure 1 The sensor component 103 in the image processing device shown in the figure can acquire the Bayer image, and the processor 102 can convert the Bayer image through a series of analog-to-digital conversions to obtain a digital image signal, that is, RAW data in the Bayer format. The RAW data here can represent unprocessed image data.

[0055] Specifically, Figure 3 As shown, the original RAW data may include multiple pixel array units, and one pixel array unit may include 2 green (green, G) pixels, 1 blue (blue, B) pixel and 1 red (red, R) pixel. Figure 3 The four pixels in the dashed box are a pixel array unit. H represents the height of the original RAW data, and W represents the width of the original RAW data. A pixel of the original RAW data has only one color, namely red, green, or blue, and each pixel has a pixel value.

[0056] Further, preprocessing the raw image data may specifically include: performing black level correction, normalization processing and channel splitting processing on the raw RAW data to obtain first RAW data; performing downsampling processing, black level correction, normalization processing and channel splitting processing on the raw RAW data to obtain second RAW data.

[0057] Among them, black level correction processing can refer to the process of restoring the pixel value range of the pixels in the original RAW data to the standard pixel value range. For example, the pixel value range of the pixels in the original RAW data is 5 to 255, and the standard pixel value range is 0 to 255. Since the image data collected by the sensor cannot provide sufficiently high accuracy when undergoing analog-to-digital conversion, it is usually impossible to convert a very small voltage value. A fixed offset needs to be added before the analog-to-digital conversion so that the lowest input level is not zero. For example, if the fixed offset is 5, the pixel value range obtained is 5 to 255. Therefore, when processing the original RAW data, the pixel values ​​of the pixels in the original RAW data can be restored to adjust the minimum value of the pixel value range to zero. This adjustment process is the black level correction processing.

[0058] Normalization processing may refer to the process of converting the pixel value range of the pixel points in the image data from 0 to 255 to 0 to 1. Normalization processing can reduce the amount of calculation of subsequent calculation processing, thereby improving the calculation efficiency of image data processing, and also facilitates the use of floating-point data calculations in subsequent image processing operations.

[0059] Channel splitting can refer to the process of splitting image data into several single-pixel channels, for example, splitting the original RAW data in the form of pixel array units (such as Figure 3 The data format shown in the figure is split into four single-pixel channels of R, G, B, and G to facilitate the subsequent processing of each single-pixel channel separately.

[0060] Downsampling processing may refer to a process of resampling the original image data according to a certain sampling coefficient to generate new image data. Specifically, the downsampling processing may adopt downsampling methods such as bilinear interpolation and bicubic interpolation. For example, the downsampling coefficient is k, which means that in the original RAW data, for each row and column of pixels, one pixel is taken every k points to resample, thereby generating new image data.

[0061] For example, Figure 4 The present invention is a flow chart of preprocessing the original RAW data. The process of obtaining the first RAW data may be: performing black level correction and normalization processing on the original RAW data, and outputting RAW floating point data, which may be data with a decimal part and a pixel value of 0 to 1; then, performing channel splitting processing on the output RAW floating point data to obtain the first RAW data, for example, the first RAW data may include image data of four channels of R, G, B, and G, the pixel value range of each pixel point is between 0 and 1, and the image height corresponding to each channel is H / 2 and the width is W / 2. Among them, the process of obtaining the second RAW data can be: first perform 8-fold downsampling on the original RAW data, output the downsampled RAW data, then perform black level correction and normalization, and output the downsampled RAW floating-point data; then, perform channel splitting on the downsampled RAW floating-point data, and output the second RAW data. For example, the second RAW data can also include image data of four channels of R, G, B, and G, and the pixel value range of each pixel is between 0 and 1, and the image height corresponding to each channel is H / 8 and the width is W / 8.

[0062] S202: Perform local pixel processing on the first RAW data to obtain first image data.

[0063] The local pixel processing may refer to processing local pixels in the first RAW data, and the local pixel processing may be used to change the image features of a specific area in the image. The local pixel processing may generally include noise reduction processing, de-mosaic processing, or local brightness processing.

[0064] In a possible implementation, performing local pixel processing on the first RAW data to obtain the first image data may specifically include: performing noise reduction processing on local pixels of the first RAW data; or performing demosaic processing on local pixels of the first RAW data; or performing noise reduction processing and demosaic processing on local pixels of the first RAW data. The processed image data may be linear RGB image data.

[0065] Among them, noise reduction processing can specifically refer to processing to eliminate or reduce noise in image data. Since the original RAW data is usually interfered by the imaging device or external environmental noise during the digital processing and data transmission process, the original RAW data and the pre-processed first RAW data and second RAW data are usually RAW data containing noise, so noise reduction processing is required. De-mosaicing can be a kind of image processing for reconstructing color, and its purpose is to reconstruct a full-color image from the input incomplete color sampling image data, that is, to reconstruct the complete RGB three primary color image data of each pixel.

[0066] RGB image data can also be called three-primary color image data, where one pixel is a mixture of three colors: red (Red, R), green (Green, G) and blue (Blue, B). R, G, B each occupy one byte, and the value range is 0 to 255; after combination, one pixel can represent 256×256×256 colors. For example, black: R=G=B=0, white: R=G=B=255, yellow: R=G=255, B=0, etc. Therefore, to process the first RAW data into an RGB image, it is necessary to interpolate the color values ​​around each pixel or fill in two other colors through calculation processing, and finally generate a color RGB image. Linear RGB image data is an image in which the change of pixel color can be represented by the linear change of pixel value data.

[0067] For example, Figure 5A process for obtaining first image data is shown. The original RAW data is input. Since there is generally noise in the input data, in order to achieve adjustable noise, a channel σ for adjusting the noise level can be added, and a series of operations for local pixel processing (for example, two-dimensional convolution operations, rectified linear unit (Rectified Linear Unit, ReLU) operations) are performed, and the output is 3-channel noise-free linear RGB data. The embodiment of the present application does not limit the specific process of local pixel processing, and general noise reduction and demosaicing are applicable to this application. By adjusting the operation parameters and operation structure in the processing process, different degrees of noise reduction and demosaicing processing effects can be achieved, the parameters of local pixel processing can be adjusted, and the flexibility and processing accuracy of image processing are improved.

[0068] It should be noted that the noise reduction processing in the embodiment of the present application can be implemented by a corresponding neural network. When the neural network used for noise reduction is trained, noise can be added to the training sample by adjusting the value of the channel σ. The value of the channel σ is the same as the added noise. After training, the corresponding relationship between the obtained neural network and the noise level is established. Therefore, when processing a noisy image, noise reduction can be performed according to the value of the channel σ.

[0069] Furthermore, performing local pixel processing on the first RAW data to obtain the first image data may specifically include: performing brightness processing on local pixels of the first RAW data to obtain a first brightness ratio matrix.

[0070] Among them, local brightness processing mainly processes the brightness information of local pixels of image data to achieve the purpose of changing the local brightness of the image. The first RAW data after channel splitting is preprocessed and subjected to local brightness processing to obtain a first brightness ratio matrix, which can also be called a first brightness enhancement ratio matrix, which is used to represent the brightness enhancement information of local pixels. Specifically, local brightness processing can extract brightness information for each pixel, or a 3x3, 5x5 pixel block, to obtain a local brightness enhancement ratio.

[0071] For example, Figure 6 This is a schematic diagram of brightness processing, where the input is a four-channel first RAW data, and the pixel values ​​of each pixel point of the four channels of R, G, B, and G can be used as four brightness matrices, and the height of each brightness matrix can be H / 2 and the width can be W / 2. These four brightness matrices are processed through a series of operations (for example, combination operations, convolution operations, and excitation function operations, etc.) to obtain a first brightness ratio matrix, and the height of the brightness ratio matrix can be H and the width can be W.

[0072] It should be noted that the specific process of brightness processing can refer to the relevant description of the prior art, and the embodiments of the present application do not specifically limit this. In the brightness processing process, by adjusting the parameters and operation structure of the operation, the brightness processing of the local image to different degrees can be achieved, thereby improving the flexibility and processing accuracy of the image processing.

[0073] In addition, in the above step S202, the first RAW data is locally pixel processed to obtain the first image data, which may also include: color enhancement or detail enhancement processing of local pixels of the first RAW data. Among them, color enhancement refers to the technology of performing color synthesis or color display through various methods and means to highlight the differences between different objects and improve the image display effect. Detail enhancement refers to the process of adjusting the details of the image, such as brightness, contrast, clarity, etc., through different methods and means, and this embodiment of the application will not be specifically elaborated on this.

[0074] S203: Perform global pixel processing on the second RAW data to obtain second image data. S202 and S203 may be performed in any order. Figure 2 The example in which S202 and S203 are executed in parallel is used for explanation.

[0075] The global pixel processing refers to processing all pixels of the image data, and the global pixel processing can be used to adjust a certain image feature of the overall image, such as the color, contrast, exposure, etc. of the overall image. Specifically, performing the global pixel processing on the second RAW data to obtain the second image data may specifically include: performing color processing on the global pixels of the second RAW data to obtain a color conversion matrix.

[0076] The color processing may be color correction (CC), which is specifically based on optical theory to accurately restore the overall color of the image to the real color tone of the shooting scene perceived by the human eye. The output color conversion matrix can be used to identify the color information of each pixel.

[0077] Exemplarily, the second RAW data includes image data of four channels of R, G, B, and G after downsampling processing, and the pixel value of each pixel point of the four channels of R, G, B, and G can be used as four color matrices, and the height of each color matrix can be H / 8 and the width can be W / 8. The four color matrices are processed through a series of operations (such as combination operations, convolution operations, and excitation function operations, etc.) to obtain a color conversion matrix, and the height of the color conversion matrix can be H and the width can be W. By processing the colors of the image at different levels globally, the flexibility and processing accuracy of the image processing can be improved.

[0078] Furthermore, in the above step S203, global pixel processing is performed on the second RAW data to obtain the second image data, which may also include: performing automatic white balance, automatic focus and other processing on the global pixels of the second RAW data. Among them, the automatic white balance processing may include adjusting the pixels that have color cast due to illumination or other reasons to their original colors through color restoration and color adjustment. The automatic focus processing may be a processing for adjusting the clarity of the key position of the image, and the automatic focus may make the focused position the clearest place in the entire image. In practical applications, the global pixel processing may also include other different processing methods, which are not specifically limited in the embodiments of the present application.

[0079] Optionally, performing global pixel processing on the second RAW data to obtain second image data may also include: performing global pixel brightness processing on the second RAW data to obtain a second brightness ratio matrix. The global brightness processing process is similar to the above-mentioned local brightness processing process, which is to adjust the brightness value of the entire image. The specific global brightness processing can extract a brightness enhancement ratio for the entire image, or each grayscale value (0 to 255) has a brightness enhancement ratio, that is, gamma correction processing. The specific processing principle can refer to the relevant description of the above-mentioned local brightness processing, and the embodiments of the present application will not be repeated here.

[0080] It should be noted that the local pixel processing and global pixel processing in the embodiments of the present application can be implemented by corresponding neural networks, and a stable neural network model is obtained by learning and training a large amount of input data. The embodiments of the present application do not limit the specific structure of the neural network. The user can set the parameters and structures of different neural networks according to the desired target image, and can adjust the color, brightness, etc. of the target image by adjusting the structure and parameters of the neural network, that is, improve the flexibility and processing accuracy of image processing.

[0081] S204: Generate target image data according to the first image data and the second image data.

[0082] Among them, the target image data can be a target RGB image displayed on the device display screen. According to the first image data and the second image data, the generation of the target image data can be realized by a processor in the image processing device; the display of the target RGB image can be specifically realized by a display panel included in the multimedia component in the image processing device.

[0083] Specifically, the target image data is generated according to the first image data and the second image data. Specifically, when the first image data includes linear RGB image data obtained by de-mosaicing and noise reduction processing and a brightness ratio matrix R obtained by local brightness processing, and the second image data includes a color conversion matrix T obtained by global color processing, the brightness ratio matrix R and the color conversion matrix T can be superimposed on the linear RGB image data to generate the target RGB image data. Exemplarily, the target RGB image data can be generated by the formula F(linear RGB)*T*R, where F represents a function that performs certain operations on the R, G, and B values ​​of the linear RGB data, which can include operations such as squaring and cross multiplication of the R, G, and B values; * represents a matrix multiplication operation.

[0084] Furthermore, the F function operates on linear RGB data, which may include squaring the R, G, and B values ​​to obtain R 2 , G 2 , B 2 , and cross-multiply to get R*B, R*G, B*G, and then R, G, B, R 2 , G 2 , B 2 , and R*B, R*G, B*G are combined in a certain order and multiplied with the color conversion matrix T to obtain the image data after color processing, and then multiplied with the brightness ratio matrix R to obtain the target RGB image.

[0085] Exemplarily, in addition to the above example, the processing method for generating target RGB image data according to the linear RGB image data, the brightness ratio matrix R and the color conversion matrix T has another embodiment of F(linear RGB*R)*T, that is, firstly performing brightness enhancement processing on the linear RGB image data, and then performing color processing to obtain the target RGB image. The embodiment of the present application does not specifically limit this.

[0086] In an embodiment of the present application, at least one local pixel processing step and at least one global pixel processing step are independent of each other and executed in parallel, and the parameters of the local pixel processing and the global pixel processing can be adjusted as needed, so that each processing step can be adjusted separately, thereby improving the flexibility and adjustability of image processing; at the same time, the image data of each local pixel and the image data of each global pixel processing are not dependent on each other, thereby retaining the original image information of the processed image to the greatest extent, improving the processing accuracy of the image, and improving the clarity of the final generated image.

[0087] The present application also provides an image processing system. Figure 7As shown, the system may include: a preprocessing circuit 701, a local pixel processing network 702, a global pixel processing network 703 and an image synthesis circuit 704. The preprocessing circuit 701 may be used to preprocess the original RAW data to obtain first RAW data and second RAW data, wherein the resolution of the image corresponding to the second RAW data is smaller than the resolution of the image corresponding to the first RAW data; the local pixel processing network 702 may be used to receive the first RAW data output by the preprocessing circuit 701, and perform local pixel processing on the first RAW data to obtain first image data; the global pixel processing network 703 may be used to receive the second RAW data output by the preprocessing circuit 701, and perform global pixel processing on the second RAW data to obtain second image data; the image synthesis circuit 704 may be used to receive the first image data output by the local pixel processing network 702 and the second image data output by the global pixel processing network 703, and generate target image data according to the first image data and the second image data.

[0088] Furthermore, the preprocessing circuit 701 can be specifically used to: perform black level correction, normalization processing and channel splitting processing on the original image data to obtain first RAW data; perform downsampling processing, black level correction, normalization processing and channel splitting processing on the original image data to obtain second RAW data.

[0089] Furthermore, the first image data may be linear RGB image data, and the local pixel processing network 702 may include a primary processing network, which is specifically used to: receive the first RAW data output by the preprocessing circuit 701, and perform at least one of noise reduction processing or demosaicing processing on local pixels of the first RAW data to obtain linear RGB image data.

[0090] Optionally, the first image data may also include a first brightness ratio matrix, and the local pixel processing network 702 may also include a first brightness processing network. The first brightness processing network may be specifically used to: receive the first RAW data output by the preprocessing circuit 701, and perform brightness processing on local pixels of the first RAW data to obtain a first brightness ratio matrix.

[0091] Furthermore, the second image data may include a color conversion matrix, and the global pixel processing network 703 may include a color processing network, which may be specifically used to: receive the second RAW data output by the preprocessing circuit 701, and perform color processing on the global pixels of the second RAW data to obtain a color conversion matrix.

[0092] Optionally, the second image data may also include a second brightness ratio matrix, and the global pixel processing network 703 may also include a second brightness processing network, which may be specifically used to: receive the second RAW data output by the preprocessing circuit 701, and perform brightness processing on the global pixels of the second RAW data to obtain a second brightness ratio matrix.

[0093] Specifically, the primary processing network, the color processing network, the first brightness processing network, and the second brightness processing network can be obtained by training an artificial neural network. An artificial neural network is an abstraction of a human brain neural network from the perspective of information processing, and a certain operation model is established. Different networks are formed according to different connection methods, and a large amount of input data can be trained to obtain a stable, parameter-adjustable operation model. The operation of the neural network may include convolution operations, linear rectification functions (Rectified Linear Unit, ReLU) and excitation functions, etc., and this application does not specifically limit this.

[0094] Exemplarily, the color processing network of the embodiment of the present application may be composed of N convolutional layers, M pooling layers, and K fully connected layers. The convolutional layer can perform feature extraction on the input data, and may contain multiple combination operations, convolution operations, and activation function operations; after the convolutional layer performs feature extraction, the output feature map will be passed to the pooling layer for feature selection and information filtering. The pooling layer may contain a preset pooling function, and its function is to replace the result of a single point in the feature map with the feature map statistics of its adjacent area; the fully connected layer is usually built at the last part of the hidden layer of the convolutional neural network. The feature map loses its 3D structure in the fully connected layer, is expanded into a vector, and is passed to the next layer through the activation function.

[0095] In a possible implementation of this embodiment, the image synthesis circuit 704 can be specifically used to: receive one of the first brightness ratio matrix output by the first brightness processing network and the second brightness ratio matrix output by the second brightness processing network, the linear RGB image data output by the primary processing network and the color conversion matrix output by the color processing network, and generate target image data according to one of the first brightness ratio matrix and the second brightness ratio matrix, the linear RGB image data and the color conversion matrix.

[0096] In an embodiment of the present application, at least one local pixel processing network and at least one global pixel processing network are independent of each other and processed in parallel, and the parameters of the local pixel processing network and the global pixel processing network can be adjusted as needed, so that each processing network can be adjusted separately, thereby improving the flexibility and adjustability of image processing; at the same time, the image data of each local pixel and the image data of each global pixel processing are not dependent on each other, thereby retaining the original image information of the processed image to the greatest extent, improving the processing accuracy of the image, and being able to improve the clarity of the final generated image.

[0097] The present application also provides an image processing device, such as Figure 8 , the device may include: a preprocessing unit 801, a pixel processing unit 802 and an image synthesis unit 803. The preprocessing unit 801 may be used to preprocess the original RAW data to obtain first RAW data and second RAW data, wherein the resolution of the image corresponding to the second RAW data is smaller than the resolution of the image corresponding to the first RAW data; the pixel processing unit 802 may be used to perform local pixel processing on the first RAW data to obtain first image data, and perform global pixel processing on the second RAW data to obtain second image data; the image synthesis unit 803 may be used to generate target image data according to the first image data and the second image data.

[0098] Furthermore, the preprocessing unit 801 can be specifically used to: perform black level correction, normalization and channel splitting processing on the original image data to obtain first RAW data; perform downsampling processing, black level correction, normalization and channel splitting processing on the original image data to obtain second RAW data.

[0099] Furthermore, the first image data may be linear RGB image data, and the pixel processing unit 802 may be specifically configured to perform at least one of noise reduction processing and demosaicing processing on local pixels of the first RAW data to obtain linear RGB image data.

[0100] Optionally, the first image data may further include a first brightness ratio matrix, and the pixel processing unit 802 may be specifically configured to: perform brightness processing on local pixels of the first RAW data to obtain a first brightness ratio matrix.

[0101] Furthermore, the second image data includes a color conversion matrix, and the pixel processing unit can be specifically used to: perform color processing on global pixels of the second RAW data to obtain a color conversion matrix.

[0102] Optionally, the second image data further includes a second brightness ratio matrix, and the pixel processing unit 802 may be specifically configured to: perform brightness processing on global pixels of the second RAW data to obtain a second brightness ratio matrix.

[0103] In a possible implementation of this embodiment, the image synthesis unit 803 may be specifically configured to generate target image data according to one of the first brightness ratio matrix and the second brightness ratio matrix, the linear RGB image data and the color conversion matrix.

[0104] In an embodiment of the present application, at least one of the processing performed by the pixel processing unit, such as brightness processing of local pixels, color processing of global pixels, and brightness processing of global pixels, are all independent and processed in parallel, and the parameters of local pixel processing and global pixel processing can be adjusted as needed, so that each processing step can be adjusted separately, thereby improving the flexibility and adjustability of image processing; at the same time, the image data of each local pixel and the image data of each global pixel processing are not dependent on each other, thereby retaining the original image information of the processed image to the greatest extent, improving the processing accuracy of the image, and improving the clarity of the final generated image.

[0105] The present application also provides an image processing device. The structure of the device can be seen in Figure 1 The device may include a memory 101 and a processor 102 coupled to the memory, the memory 101 stores instructions and data, and the processor 102 runs the instructions in the memory 101. When the processor 102 runs the stored instructions, the device can execute the image processing method provided by steps S201 to S204 in the above method embodiment.

[0106] In the several embodiments provided in the present application, it should be understood that the disclosed methods, systems and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0108] In addition, each functional unit in each embodiment of the present application may be integrated into a data processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0109] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0110] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Preprocessing the original RAW data to obtain first RAW data and second RAW data, wherein the resolution of an image corresponding to the second RAW data is smaller than the resolution of the image corresponding to the first RAW data; Performing local pixel processing on the first RAW data to obtain first image data, and performing global pixel processing on the second RAW data to obtain second image data; Target image data is generated based on the first image data and the second image data.

2. The method according to claim 1, characterized in that: The first image data includes linear RGB image data, and the performing local pixel processing on the first RAW data to obtain the first image data includes: At least one of noise reduction processing and demosaicing processing is performed on local pixels of the first RAW data to obtain the linear RGB image data.

3. The method according to claim 1 or 2, characterized in that: The first image data further includes a first brightness ratio matrix, and the performing local pixel processing on the first RAW data to obtain the first image data further includes: Performing brightness processing on local pixels of the first RAW data to obtain the first brightness ratio matrix.

4. The method according to claim 1 or 2, characterized in that: The second image data includes a color conversion matrix, and the performing global pixel processing on the second RAW data to obtain the second image data includes: Performing color processing on global pixels of the second RAW data to obtain the color conversion matrix.

5. The method according to claim 1 or 2, characterized in that: The second image data further includes a second brightness ratio matrix, and the performing global pixel processing on the second RAW data to obtain the second image data includes: Perform brightness processing on the global pixels of the second RAW data to obtain the second brightness ratio matrix.

6. The method according to claim 5, characterized in that Generating target image data according to the first image data and the second image data includes: The target image data is generated according to one of a first brightness ratio matrix and a second brightness ratio matrix, linear RGB image data and a color conversion matrix; wherein the first brightness ratio matrix is ​​obtained by performing brightness processing on local pixels of the first RAW data, the linear RGB image data is obtained by performing at least one of noise reduction processing or demosaicing processing on local pixels of the first RAW data, and the color conversion matrix is ​​obtained by performing color processing on global pixels of the second RAW data.

7. The method according to claim 1 or 2, characterized in that: The preprocessing of the original RAW data to obtain the first RAW data and the second RAW data includes: Performing black level correction, normalization processing, and channel splitting processing on the original RAW data to obtain the first RAW data; The original RAW data is subjected to downsampling processing, black level correction, normalization processing, and channel splitting processing to obtain the second RAW data.

8. An image processing system, characterized in that: The system comprises: A preprocessing circuit, used for preprocessing the original RAW data to obtain first RAW data and second RAW data, wherein the resolution of the image corresponding to the second RAW data is smaller than the resolution of the image corresponding to the first RAW data; a local pixel processing network, configured to receive the first RAW data output by the preprocessing circuit, and perform local pixel processing on the first RAW data to obtain first image data; a global pixel processing network, configured to receive the second RAW data output by the preprocessing circuit, and perform global pixel processing on the second RAW data to obtain second image data; An image synthesis circuit is used to receive the first image data output by the local pixel processing network and the second image data output by the global pixel processing network, and generate target image data according to the first image data and the second image data.

9. The system according to claim 8, characterized in that The first image data is linear RGB image data, and the local pixel processing network includes a primary processing network, which is specifically used for: The first RAW data output by the preprocessing circuit is received, and at least one of noise reduction processing and demosaicing processing is performed on local pixels of the first RAW data to obtain the linear RGB image data.

10. The system according to claim 8 or 9, characterized in that The first image data further includes a first brightness ratio matrix, and the local pixel processing network further includes a first brightness processing network, wherein the first brightness processing network is specifically used for: The first RAW data output by the preprocessing circuit is received, and brightness processing is performed on local pixels of the first RAW data to obtain the first brightness ratio matrix.

11. The system according to claim 8 or 9, characterized in that: The second image data includes a color conversion matrix, and the global pixel processing network includes a color processing network, and the color processing network is specifically used for: The second RAW data output by the preprocessing circuit is received, and color processing is performed on global pixels of the second RAW data to obtain the color conversion matrix.

12. The system according to claim 8 or 9, characterized in that The second image data further includes a second brightness ratio matrix, and the global pixel processing network further includes a second brightness processing network, and the second brightness processing network is specifically used for: The second RAW data output by the preprocessing circuit is received, and brightness processing is performed on global pixels of the second RAW data to obtain the second brightness ratio matrix.

13. The system according to claim 12, characterized in that The image synthesis circuit is specifically used for: Receiving one of a first brightness ratio matrix output by a first brightness processing network and a second brightness ratio matrix output by a second brightness processing network, linear RGB image data output by a primary processing network, and a color conversion matrix output by a color processing network, and generating the target image data according to one of the first brightness ratio matrix and the second brightness ratio matrix, the linear RGB image data, and the color conversion matrix; Among them, the first brightness ratio matrix is ​​obtained by the first brightness processing network performing brightness processing on local pixels of the first RAW data, the linear RGB image data is obtained by the primary processing network performing at least one of noise reduction processing or demosaicing processing on local pixels of the first RAW data, and the color conversion matrix is ​​obtained by the color processing network performing color processing on global pixels of the second RAW data.

14. The system according to claim 8 or 9, characterized in that The preprocessing circuit is specifically used for: Performing black level correction, normalization processing, and channel splitting processing on the original RAW data to obtain the first RAW data; The original RAW data is subjected to downsampling processing, black level correction, normalization processing, and channel splitting processing to obtain the second RAW data.

15. An image processing device, characterized in that: The device comprises: A preprocessing unit, configured to preprocess the original RAW data to obtain first RAW data and second RAW data, wherein the resolution of an image corresponding to the second RAW data is smaller than the resolution of the image corresponding to the first RAW data; a pixel processing unit, configured to perform local pixel processing on the first RAW data to obtain first image data, and perform global pixel processing on the second RAW data to obtain second image data; An image synthesis unit is used to generate target image data according to the first image data and the second image data.

16. The device according to claim 15, characterized in that The first image data is linear RGB image data, and the pixel processing unit is specifically used for: At least one of noise reduction processing and demosaicing processing is performed on local pixels of the first RAW data to obtain the linear RGB image data.

17. The device according to claim 15 or 16, characterized in that The first image data also includes a first brightness ratio matrix, and the pixel processing unit is further specifically configured to: Performing brightness processing on local pixels of the first RAW data to obtain the first brightness ratio matrix.

18. The device according to claim 15 or 16, characterized in that The second image data includes a color conversion matrix, and the pixel processing unit is further specifically used for: Performing color processing on global pixels of the second RAW data to obtain the color conversion matrix.

19. The device according to claim 15 or 16, characterized in that The second image data further includes a second brightness ratio matrix, and the pixel processing unit is further specifically configured to: Perform brightness processing on the global pixels of the second RAW data to obtain the second brightness ratio matrix.

20. The device according to claim 19, characterized in that The image synthesis unit is specifically used for: The target image data is generated according to one of a first brightness ratio matrix and a second brightness ratio matrix, linear RGB image data and a color conversion matrix; wherein the first brightness ratio matrix is ​​obtained by performing brightness processing on local pixels of the first RAW data, the linear RGB image data is obtained by performing at least one of noise reduction processing or demosaicing processing on local pixels of the first RAW data, and the color conversion matrix is ​​obtained by performing color processing on global pixels of the second RAW data.

21. The device according to claim 15 or 16, characterized in that The pre-processing unit is specifically used for: Performing black level correction, normalization processing, and channel splitting processing on the original RAW data to obtain the first RAW data; The original RAW data is subjected to downsampling processing, black level correction, normalization processing, and channel splitting processing to obtain the second RAW data.

22. An image processing device, characterized in that: The device comprises: a memory and a processor coupled to the memory, the memory stores instructions and data, and the processor runs the instructions in the memory so that the processor executes the image processing method according to any one of claims 1 to 7.

23. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the image processing method according to any one of claims 1 to 7.

24. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the image processing method according to any one of claims 1 to 7.

Images