Image processing apparatus, image processing method, and display apparatus

Abstract

An image processing device (100), an image processing method and a display device (700) relate to the technical field of display and digital video processing and analysis. The image processing apparatus (100) comprises: a processing module (101) configured to acquire brightness data and color data of a display image; and a calculation module (102) configured to perform convolution operation on the brightness data by using a neural network to obtain brightness enhancement data; wherein the processing module (101) is further configured to perform enhancement processing on the brightness data by using the brightness enhancement data to obtain target brightness data, and convert the target brightness data and the color data into a target image, the target image being used for outputting to a display module for display.

Description

Image processing apparatus, image processing method and display apparatus TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of display and digital video processing and analysis, and in particular to an image processing apparatus, an image processing method and a display apparatus. BACKGROUND

[0002] The principle of Organic Light Emitting Diode (OLED) display is that organic semiconductor materials and light-emitting materials emit light through carrier injection and recombination under electric field driving. In the OLED light-emitting process, the resistance existing in the circuit will generate heat, and the OLED light-emitting will also cause the surface of the display module to be heated by light radiation.

[0003] For the OLED display module, the higher the power consumption in the circuit, the more serious the heat generation. The heat generated by high power consumption accumulates and causes the temperature of the module to rise. When the temperature is too high, the electronic device will be quickly consumed, reducing the service life, thereby affecting the image display quality of the OLED display module.

[0004] SUMMARY

[0005] The present disclosure provides an image processing apparatus, an image processing method and a display apparatus.

[0006] According to a first aspect, the present disclosure provides an image processing apparatus, comprising: a processing module configured to obtain luminance data and color data of a video frame image; and a calculation module configured to perform convolution operation on the luminance data by using a neural network to obtain luminance enhancement data; wherein the processing module is further configured to perform enhancement processing on the luminance data by using the luminance enhancement data to obtain target luminance data, and convert the target luminance data and the color data into a target image, the target image being used for output to a display module for display.

[0007] According to a second aspect, the present disclosure provides an image processing method, comprising: obtaining luminance data and color data of a video frame image; performing convolution operation on the luminance data by using a neural network to obtain luminance enhancement data; performing enhancement processing on the luminance data by using the luminance enhancement data to obtain target luminance data; and converting the target luminance data and the color data into a target image, the target image being used for output to a display module for display.

[0008] According to a third aspect, the present disclosure provides a display apparatus, comprising: a display module; and an image processing apparatus provided by an embodiment of the present disclosure, configured to output a target image based on a display image required to be displayed by the display module; wherein the display module displays the target image.

[0009] According to a fourth aspect, the present disclosure provides another display device, comprising: a processor configured to perform the image processing method provided by the present disclosure; and a display module configured to display a target image output by the processor as a target image. BRIEF DESCRIPTION OF DRAWINGS

[0010] FIG. 1 is a structural schematic diagram of an image processing device according to an embodiment of the present disclosure;

[0011] FIG. 2 is a structural schematic diagram of an image processing device according to another embodiment of the present disclosure;

[0012] FIG. 3A is a working timing diagram of an image processing device according to an example;

[0013] FIG. 3B is a working timing diagram of an image processing device according to an embodiment of the present disclosure;

[0014] FIG. 4 is a structural schematic diagram of an image processing device according to another embodiment of the present disclosure;

[0015] FIG. 5 is a structural schematic diagram of an image processing device according to another embodiment of the present disclosure;

[0016] FIG. 6 is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0017] FIG. 7 is a structural schematic diagram of a display device according to an embodiment of the present disclosure; and

[0018] FIG. 8 is a structural schematic diagram of a display device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] To make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all. Based on the described embodiments of the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without any creative effort fall within the protection scope of the present disclosure. It should be noted that throughout the drawings, the same elements are denoted by the same or similar reference numerals. In the following description, some specific embodiments are only for the purpose of description, and should not be understood as any limitation on the present disclosure, but only as examples of the embodiments of the present disclosure. When it is possible to cause confusion to the understanding of the present disclosure, the conventional structures or configurations will be omitted. It should be noted that the shapes and sizes of the components in the drawings do not reflect the true size and ratio, but only illustrate the content of the embodiments of the present disclosure.

[0020] Unless otherwise defined, technical terms or scientific terms used in the present disclosure shall have the ordinary meaning understood by a person skilled in the art. The terms "first", "second", and similar terms are not intended to denote any order, quantity, or importance, but are merely used to distinguish different constituent parts.

[0021] Due to the self-luminous characteristic of OLED, the power consumption in the pixel circuit is generated only when the pixel point in the OLED display device is lighted. When displaying video and image with high brightness, the high power consumption of the pixel circuit can cause the display module to generate serious heat. Long-time high power consumption operation can cause the temperature of the display module to be overheated, thereby damaging the OLED light-emitting device and shortening the service life of the display device in the display module. Therefore, reducing the display power consumption of the display module is beneficial to prolong the service life of the display module.

[0022] Based on the possible display defects of the OLED display device, the display brightness of the display module is optimized in real time in the present disclosure. While ensuring the display quality, the average display brightness of the display module is reduced, thereby reducing the display power consumption of the display module. After reducing the high power consumption and high heat caused by the high brightness of the image in the OLED display module, the aging rate of the display device can be reduced, and the service life of the OLED display module can be prolonged.

[0023] FIG. 1 is a structural schematic diagram of an image processing device according to an embodiment of the present disclosure.

[0024] As shown in FIG. 1, the image processing device 100 includes a processing module 101 and a calculation module 102.

[0025] In the present disclosure, the processing module 101 obtains brightness data and color data of a display image. For example, the display image can be a video frame image in a frame of a video displayed by a display module. The processing module 101 extracts the brightness data and the color data from the video frame image before the display module displays the video frame image.

[0026] In the present disclosure, the brightness data can be the brightness value of all pixel points in the display image, and the color data can be the color of all pixel points in the display image. For example, the brightness data can be the Y value in the YUV color space, which represents the brightness of the pixel point, for example, the gray value of the pixel point. For example, the color data can be the UV value in the YUV color space, which represents the chroma of the pixel point.

[0027] In the present disclosure, the calculation module 102 performs convolution operation on the brightness data by using a neural network to obtain brightness enhancement data. By performing convolution operation on the brightness data of the display image by using the neural network, the calculation module 102 can extract the brightness distribution characteristics of the display image, so that the brightness of the image data can be adjusted based on the brightness distribution characteristics.

[0028] For example, brightness enhancement data is brightness mapping information adapted to the displayed image. For instance, brightness enhancement data could be a brightness look-up table (LUT). Based on the brightness LUT, a mapping value can be determined for each Y value in the displayed image, where the mapping value is the adjusted Y value of the pixel.

[0029] In this embodiment of the disclosure, the processing module 101 uses the brightness enhancement data to enhance the brightness data to obtain the target brightness data, and converts the target brightness data and color data into a target image, which is then output to the display module for display.

[0030] For example, the target brightness data can be the brightness value after mapping brightness data using a luminance LUT. Mapping brightness data using a luminance LUT can highlight useful information in an image, enhance the image's light contrast, and reduce the average brightness of the displayed image. The average brightness can be the average of the brightness values ​​of all pixels in the displayed image. For example, after mapping with a luminance LUT, the brightness of some pixels in the displayed image increases, while the brightness of some pixels decreases, but the average brightness of the displayed image decreases. Reducing the average brightness of the displayed image also reduces the power consumption of the display module during the display process, thereby mitigating heat generation during display.

[0031] In this embodiment, the calculation module 102 uses a neural network to perform convolution operations on the brightness data to extract features of brightness information in the displayed image, so that the brightness enhancement data obtained by the calculation module 102 matches the brightness distribution characteristics of the displayed image. After mapping the brightness data of the displayed image based on the brightness enhancement data, the average brightness of the displayed image decreases, but the display quality of the displayed image is not affected.

[0032] For example, the calculation module 102 can use a neural network to determine the image category of the displayed image based on its brightness data. Based on the image category, corresponding brightness enhancement data is determined. For example, image categories may include bright scene image categories, dark scene image categories, landscape image categories, portrait image categories, blue image categories, and red image categories, etc. Each image category corresponds to its own brightness enhancement data. The image categories can be preset, or the brightness enhancement data for each image category can be pre-trained.

[0033] For example, the computing module 102 can determine the image category to which the display image belongs from a plurality of preset image categories based on the luminance data of the display image by using the neural network. For example, the computing module 102 can determine the weight of the display image relative to the plurality of preset image categories by using the neural network, and generate new luminance enhancement data based on the respective weights of the plurality of preset image categories and the luminance enhancement data. The new luminance enhancement data covers the luminance enhancement data corresponding to the plurality of preset image categories.

[0034] For example, the video that the display module needs to display includes a plurality of video frame images. In the process of displaying the video by the display module, the processing module 101 can obtain the luminance data of the plurality of video frame images in real time in sequence, and the computing module 102 can generate respective luminance enhancement data based on the luminance data of the plurality of video frame images in sequence. The processing module 101 can determine the target luminance data of the plurality of video frame images according to the respective luminance enhancement data of the plurality of video frame images, and output the video frame images after luminance adjustment in sequence, so that the luminance adjustment of the frame image is completed before the display module displays the video frame image, and the real-time optimization of the display image of the display module is realized.

[0035] The process of displaying the image by the display module and the process of adjusting the video frame image by the image processing apparatus 100 can be performed synchronously. For example, when the display module displays the first frame image, the image processing apparatus 100 adjusts the luminance of the second frame image. When the display module displays the second frame image, the image processing apparatus 100 adjusts the luminance of the third frame image. It should be noted that the speed of displaying the image by the display module and processing the image by the image processing apparatus 100 is not limited in the present disclosure, and the synchronous processing scenario described above is only illustrative.

[0036] In the embodiments of the present disclosure, the image processing apparatus 100 can be located in a display driver chip (DDIC) of a display device. For example, the image processing apparatus 100 can be implemented in the form of a field programmable gate array (FPGA) in a data processor (Data Processor) of the display driver chip.

[0037] For example, the display driving chip can receive video data through a Mobile Industry Processor Interface (MIPI) and a Serial Peripheral Interface (SPI). The received video data can be decompressed and sent to the data processor, or the received data can be cached in a graphic RAM (GRAM). The data processor can read video frame data that needs to be processed from the GRAM.

[0038] The data processor performs conversion, scaling, color correction, and the like on the video frame images in the received video data. For example, the data processor can improve the brightness deviation of the video frame images through a Demura function. For example, the data processor receives instructions from a timing controller (TCON) to perform format conversion, brightness adjustment, contrast adjustment, and color adjustment, and the like on the video frame images. For example, the data processor can also perform current-resistance voltage drop compensation on the supply voltage ELVDD to avoid brightness differences in the data signal after the video frame image conversion data is written into the pixel circuit. For example, the data processor can also perform gamma correction on the video frame data to perform color correction, brightness adjustment, and contrast adjustment, and the like on the video frame images.

[0039] For example, the display image obtained by the image processing apparatus 100 can be a video frame image that has undergone gamma correction, demura correction, color correction, contrast adjustment, brightness adjustment, format conversion, and the like. The image processing apparatus 100 performs brightness enhancement processing on the received video frame, so that the average brightness of the video frame image is reduced.

[0040] For example, the target image output by the image processing apparatus 100 can be an RGB pixel value or a grayscale value, and the like. The image processing apparatus 100 sends the target image to a source driver IC. The source driver power supply converts the target image into a data signal based on a timing control signal provided by a timing controller. The data signal is written into the pixel circuit of the display module, so that the target image is displayed through the display module.

[0041] In the embodiments of the present disclosure, the brightness of the display image is analyzed by the calculation module 102 using a neural network to determine the image category of the display image, so that the brightness enhancement data of the display image can be determined, the brightness of the display image can be accurately adjusted, and the average brightness of the display image can be reduced. When the display module displays the display image after brightness adjustment, the display power consumption is reduced, and the negative impact of display power consumption on the display device is improved.

[0042] FIG. 2 is a structural schematic diagram of an image processing apparatus according to another embodiment of the present disclosure.

[0043] As shown in FIG. 2, the image processing apparatus 200 includes a processing module 201 and a calculation module 202.

[0044] In the embodiment of the present disclosure, the processing module 201 includes a first converter 211, a second converter 213, a delay device 212, a first generator 214 and an enhancer 215.

[0045] In the embodiment of the present disclosure, the first converter 211 converts pixel values of a display image into luminance data and color data. The second converter 213 converts target luminance data and color data into pixel values of a target image.

[0046] For example, the pixel values are three primary color (RGB) pixel values of the display image. The first converter 211 converts the RGB values of the display image into YUV values. The target luminance data is Y value after luminance adjustment. The second converter 213 converts the YUV values after luminance adjustment into RGB values and outputs to a display module. The RGB values output by the second converter 213 form a target image, and the display module displays the target image.

[0047] For example, the first converter 211 can convert the RGB values into YUV values based on formula (1):

[0048] wherein p_ry, p_gy, p_by, p_ru, p_gu, p_bu, p_rv, p_gv and p_bv are RGB-YUV conversion coefficients, r, g and b are R value, G value and B value respectively, y is Y value, u is U value, v is V value, and offset is offset value. The offset value is related to the value range of r, g and b. For example, when the value range of r, g and b is 0-255, the offset can be 128. For example, when the value range of r, g and b is converted to 0-1, the offset can be 0.5.

[0049] According to the color standard of the image required to be displayed by the display module, corresponding conversion coefficients can be selected. For example, the color standard can include BT601, BT709 and BT2020, and the RGB-YUV conversion coefficients of the color standards BT601, BT709 and BT2020 are shown in Table 1:

[0050] Table 1

[0051] For example, the second converter 213 can convert the YUV values into RGB values based on formula (2):

[0052] wherein p_a, p_b, p_c and p_d are YUV-RGB conversion coefficients.

[0053] YUV-RGB conversion coefficients of color standards BT601, BT709 and BT2020 are shown in Table 2:

[0054] Table 2

[0055] The first converter 211 and the second converter 213 convert based on the same color standard.

[0056] In order to simplify the calculation complexity, the parameters in formula (1) and formula (2) are stored and calculated in unsigned integer. Since unsigned integer is used for calculation, in order to prevent negative numbers from appearing when offset is subtracted, and subsequent multiplication calculation of negative numbers, formula (2) can be adjusted to formula (3), thereby simplifying the calculation complexity:

[0057] For example, the first converter 211 and the second converter 213 can be implemented based on FPGA. Based on formula (3), YUV can be converted to RGB, which can simplify the operation of FPGA.

[0058] In the embodiment of the present disclosure, since the calculation amount of the calculation module 102 is large and the calculation process is complex, in order to reduce the consumption of hardware resources, the calculation module 102 can use a high-frequency clock for calculation to improve the calculation efficiency of the calculation module 102. Based on the calculation of the calculation module 102 based on the high-frequency clock, it can be ensured that the luminance data of an input frame of display image can be calculated within a limited frame time, which avoids the calculation flow congestion caused by too long local calculation delay time, and also avoids the calculation error of the whole data of the calculation module 102.

[0059] In the embodiment of the present disclosure, the processing module 102 can process the display image based on a regular frequency processing clock, and cache the data obtained by processing. For example, the processing module 101 can process the display data based on a first processing clock, and the calculation module 202 can perform convolution operation based on a second processing clock, and the frequency of the second processing clock is greater than that of the first processing clock. For example, the frequency of the first processing clock can be 148.5M, and the frequency of the second processing clock can be 300M.

[0060] The processing module 101 processes the display image with the 148.5M video clock as a processing clock to obtain the luminance data, and can write the luminance data Y into a storage unit, such as a double date rate (DDR) memory. When calculating the luminance enhancement data YLUT, the calculation module 202 reads the luminance data Y from the storage unit based on the 300M high-frequency clock for subsequent calculation.

[0061] The calculation module 202 needs to consume a certain amount of time to perform convolution operation on the luminance data Y using a convolutional neural network (CNN). Therefore, compared with the color data, the target luminance data Y' of the same display image arrives at the second converter 213 later than the color data U / V, which causes the second converter 213 to make a data processing error due to asynchronization of the color data and the target luminance data Y'.

[0062] In the embodiment of the present disclosure, the delay device 212 delays the output of the color data to the second converter 213, so that the second converter 213 synchronously receives the target luminance data and the color data.

[0063] For example, the delay device 212 can write the color data U / V into a storage unit, and read the color data U / V from the storage unit at a target time and output to the second converter 213. The target time is determined based on the time when the second converter 213 receives the target luminance data Y'.

[0064] In the embodiment of the present disclosure, the storage unit can be a DDR. After the first converter 211 outputs the luminance data Y and the color data U / V, the delay device 212 writes the color data into the DDR, delaying the time when the color data U / V is input to the second converter 213. The delay device 212 reads the color data U / V from the DDR at a target time and sends it to the second converter 213, so that the color data U / V can arrive at the second converter 213 synchronously with the target luminance data Y'. For example, the target time can be the time when the enhancer 215 outputs the target luminance data Y'.

[0065] The function of the delay device 212 is exemplarily described in combination with FIG. 3A and FIG. 3B.

[0066] FIG. 3A is a working timing diagram of an image processing apparatus according to one example, and FIG. 3B is a working timing diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0067] For example, the calculation module 202 needs to consume one frame processing time to calculate the brightness enhancement data, and the calculation module 202 outputs the brightness enhancement data with a two-frame delay relative to the color data of the frame display image output by the processing module 201. An example of the output timing of the color data and the target brightness data of the display image is shown in FIG. 3A.

[0068] As shown in FIG. 3A, during the processing of the first frame image, the first converter 211 outputs the brightness data Y1 to the first generator 214 and outputs the color data U1 / V1 to the second converter 213. The target brightness data Y1’ is output through the calculation module 202 and the enhancer 215. Since the calculation process of the calculation module 202 needs to consume one frame processing time, when the calculation module 202 calculates the target brightness data Y1’ of the first frame image, the processing module 201 can simultaneously extract the brightness data Y2 and the color data U2 / V2 of the second frame image and the brightness data Y3 and the color data U3 / V3 of the third frame image. When the calculation module 202 outputs the target brightness data Y1’ of the first frame image, the processing module 201 outputs the color data U3 / V3 of the third frame image. At this time, the image processing apparatus 200 synchronously outputs the target brightness data Y1’ of the first frame image and the color data U3 / V3 of the third frame image. This causes the color data and the target brightness data of each frame image to be out of synchronization, and the color data of each frame image reaches the second converter 213 earlier than the target brightness data, which causes the second converter 213 to have a data processing error due to the data out of synchronization.

[0069] Based on the above problem, the present disclosure delays the color data of each frame image to be input to the second converter 213 based on the delay device 212, so that the color data and the target brightness data of the same frame image are synchronously input to the second converter 213. The output timing of the color data and the target brightness data of the display image of the present disclosure is shown in FIG. 3B.

[0070] As shown in FIG. 3B, during the processing of the first frame of image, the first converter 211 outputs the luminance data Y1 to the first generator 214 and outputs the color data U1 / V1 to the delay device 211. The delay device 211 can write the color data U1 / V1 into a storage unit. When the target luminance data Y1' is outputted through the calculation module 202 and the enhancer 215, the delay device 212 outputs the color data U1 / V1 in the storage unit. At this time, the image processing apparatus 200 synchronously outputs the target luminance data Y1' and the color data U1 / V1 of the first frame of image. Correspondingly, the image processing apparatus 200 synchronously outputs the target luminance data Y2' and the color data U2 / V2 of the second frame of image, and the image processing apparatus 200 synchronously outputs the target luminance data Y3' and the color data U3 / V3 of the third frame of image. By delaying the output of the color data of each frame of image to the second converter 213 through the delay device 212, the synchronous output of the color data and the target luminance data of each frame of image to the second converter is ensured, and the data processing error caused by the asynchronization of data is reduced.

[0071] Referring back to FIG. 2, the first converter 211 converts the pixel value RBG of the display image into the luminance data Y and the color data U / V. The first generator 214 generates the luminance feature map F based on the luminance data Y, and the luminance feature map F is subjected to convolution operation by the calculation module 202 to obtain the luminance enhancement data YLUT. The enhancer 215 performs enhancement processing on the luminance data Y based on the luminance enhancement data YLUT to obtain the target luminance data Y'. The delay device 212 delays the output of the color data U / V to the second converter 213. For example, the delay device 212 can delay the color data U / V by a corresponding number of clocks by using a register to obtain the delayed color data U' / V'. The value of the delayed color data U' / V' is the same as that of the color data U / V, and the delayed color data U' / V' is clocked synchronously with the luminance enhancement data Y', so that the second converter 213 can synchronously receive the delayed color data U' / V' and the luminance enhancement data Y', thereby correctly completing the conversion from YUV to RGB and outputting the pixel value RGB' of the target image.

[0072] In the embodiments of the present disclosure, the enhancer 215 performs mapping enhancement on the luminance data Y by using the luminance enhancement data LUT. For example, the luminance data Y is taken as the table lookup address of the luminance lookup table, and the output data determined based on the table lookup address is taken as the target luminance data Y'.

[0073] Since the data amount of the display image is inconsistent with the data amount of the input data supported by the CNN, the first generator 214 can process the luminance data Y to generate the luminance feature map F, and the data amount of the luminance feature map F conforms to the data amount of the input data supported by the CNN.

[0074] In the embodiments of the present disclosure, the first generator 214 partitions the luminance data Y based on the data amount of the input data supported by the neural network, to obtain a plurality of partitions, and the luminance data Y includes luminance values of all pixel points in the display image. The first generator 214 generates the luminance feature map F by using the luminance value statistics of a plurality of pixel points in each partition of the plurality of partitions.

[0075] For example, when the data amount of the luminance data Y is greater than the data amount of the input data supported by the neural network, the first generator 214 can perform partition downsampling on the luminance data Y based on the data amount of the input data supported by the neural network, to reduce the data amount of each partition and thereby reduce the size of the luminance data Y.

[0076] The downsampling can simplify the luminance data Y while retaining the luminance features in the luminance data Y. The downsampling can reduce the data amount of the luminance data Y, reduce the calculation amount of the neural network, prevent overfitting in the convolution operation, and increase the receptive field of the neural network, so that the convolution kernel can obtain global information of the luminance data Y.

[0077] In the embodiments of the present disclosure, the luminance value statistics of a plurality of pixel points in each partition include the maximum luminance value and the average luminance value of the luminance values of all pixel points in each partition. The first generator 214 determines the partition luminance value of each partition based on the maximum luminance value and the average luminance value of the plurality of pixel points in each partition, and generates the luminance feature map F according to the partition luminance value of each partition.

[0078] For example, the resolution of the display image can be 7680x4320, and the resolution of the luminance data Y is also 7680x4320. The resolution of the input data required by the CNN is 480x270, so the first generator 214 can perform 16 times downsampling on the luminance data Y to obtain the luminance feature map F.

[0079] The first generator 214 divides the 7680x4320 luminance data Y into 480x270 partitions, each partition including luminance values of 16x16 pixel points, and combines the luminance values of 16x16 pixel points in each partition into one luminance value to obtain a partition luminance value. For example, the first generator 214 determines the maximum luminance value and the average luminance value of all pixel points in each partition, and reduces the data amount of the luminance data Y based on the partition mean-max weighted method.

[0080] For example, the first generator 214 can generate the partition luminance value F of each partition based on formula (4) out :

[0081] wherein Y avg is the average luminance value, and Ymax is the maximum luminance value, blocki represents the ith partition, Y i is the luminance value of a pixel point in the ith partition, μ max and μ avg are pre-input normalized weight parameters, respectively representing the weight of the maximum luminance value and the weight of the average luminance value. F out is a partition luminance value of each partition in the luminance feature map, and the luminance feature map includes 480x270 partition luminance values F out .

[0082] FIG. 4 is a structural schematic diagram of an image processing apparatus according to another embodiment of the present disclosure.

[0083] As shown in FIG. 4, the image processing apparatus 400 includes a processing module 401 and a calculation module 402.

[0084] In the embodiment of the present disclosure, the processing module 401 includes a first converter 411, a second converter 413, a delay device 412, a first generator 414 and an enhancer 415. The calculation module 402 includes a data reader / writer 421, a calculator 422 and a second generator 423.

[0085] In the embodiment of the present disclosure, the first converter 411, the second converter 413, the delay device 412, the first generator 414 and the enhancer 415 are similar to the first converter 211, the second converter 213, the delay device 212, the first generator 214 and the enhancer 215 described above, and are not described again for simplicity.

[0086] In the embodiment of the present disclosure, the data reader / writer 411 can read the luminance data from the storage unit through a storage unit interface DDRIO, and the luminance data can be the luminance feature F after scaling processing. The data reader / writer 421 can also receive the luminance feature map F from the first generator 414 and send the luminance feature map F to the calculator 422. For example, the data reader / writer 411 can be a direct memory access (DMA).

[0087] In the embodiment of the present disclosure, the calculator 422 performs convolution operation on the luminance data by using a plurality of convolution kernels kernel of a neural network to obtain a plurality of weight data Q. For example, the calculator 422 performs convolution operation on the luminance feature map F by using a CNN to determine the weight of the luminance feature map F relative to a plurality of preset image categories.

[0088] For example, the CNN can be a network including 6 convolution layers, and the network structure of the CNN can be as shown in Table 3:

[0089] Table 3

[0090] As shown in Table 3, the first layer convolutional layer C1 includes 1 input channel and 16 output channels, the input data is 1 feature map of 480x270, the output data is 16 feature maps of 240x135, and the size of the convolution kernel is 3x3. The second layer convolutional layer C2 includes 16 input channels and 16 output channels, the input data is 16 feature maps of 240x135, the output data is 16 feature maps of 120x68, and the size of the convolution kernel is 3x3. The third layer convolutional layer C3 includes 16 input channels and 16 output channels, the input data is 16 feature maps of 120x68, the output data is 16 feature maps of 60x34, and the size of the convolution kernel is 3x3. The fourth layer convolutional layer C4 includes 16 input channels and 16 output channels, the input data is 16 feature maps of 60x34, the output data is 16 feature maps of 30x17, and the size of the convolution kernel is 3x3. The fifth layer convolutional layer C5 includes 16 input channels and 16 output channels, the input data is 16 feature maps of 30x17, the output data is 16 feature maps of 15x9, and the size of the convolution kernel is 3x3. The sixth layer convolutional layer C6 includes 16 input channels and 8 output channels, the input data is 16 feature maps of 15x9, the output data is 8 weight data, and the size of the convolution kernel is 15x9.

[0091] The calculator 422 can obtain the convolution kernel required by each layer of the convolutional layer from the convolution kernel parameter management unit. The number of channels of the convolution kernel required by each layer of the convolutional layer is the same as the number of input channels, and the number of convolution kernels is the same as the number of output channels. The calculator 422 extracts the luminance features in the luminance feature map F step by step by using multiple layers of convolutional layers to obtain weight values representing the probability of the preset image category. For example, the image processing apparatus 400 can store 8 preset luminance lookup tables Pre-LUT for image categories that are obtained by pre-training, and each preset luminance lookup table Pre-LUT includes mapping values for adjusting the luminance of images of the corresponding image category.

[0092] In the embodiments of the present disclosure, the calculator 422 outputs the weight data Q of the 8 preset luminance lookup tables Pre-LUT. The second generator 423 weights and combines the multiple preset luminance lookup tables Pre-LUT by using the weight data Q to obtain luminance enhancement data YLUT, and the luminance enhancement data YLUT is a luminance lookup table matching the image category of the display image. The luminance data of the display image is mapped based on the luminance enhancement data YLUT to achieve luminance enhancement.

[0093] For example, the second generator 423 can determine the luminance enhancement data YLUT according to formula (5):

[0094] Wherein, pre_LUT(i) represents the i-th preset luminance lookup table, Weight(i) is the weight data of the i-th preset luminance lookup table, and the luminance enhancement data YLUT is obtained by weighting calculation on the eight preset luminance lookup tables pre-LUT.

[0095] The second generator 423 sends the luminance enhancement data YLUT to the enhancer 415, and the enhancer 415 performs enhancement mapping on the luminance data Y based on the luminance enhancement data YLUT to obtain the target luminance data Y'.

[0096] In the embodiment of the present disclosure, since the CNN has a large amount of calculation and consumes a large amount of hardware resources on the FPGA chip, in order to reduce resource consumption, the calculator 422 adopts a serial calculation strategy to implement the convolution operation of the six-layer CNN. For example, the calculator 422 is multiplexed, and the calculator 422 is used to sequentially calculate the six-layer convolution operation in the CNN.

[0097] In the embodiment of the present disclosure, the neural network can include M cascaded convolution layers, the data reader / writer 421 and the calculator 422 perform convolution operation of the m-th convolution layer in the M convolution layers, 1≤m≤M, m and M are positive integers.

[0098] In the embodiment of the present disclosure, in response to determining that m = 1, the data reader / writer 421 reads the luminance data Y as the first input data of the first convolution layer. The calculator 422 performs convolution operation of the first convolution layer based on the first input data based on the first input data to obtain the first output data.

[0099] In response to determining that 1 < m < M, the data reader / writer 421 reads the (m-1)th input data of the (m-1)th convolution layer as the mth input data of the mth convolution layer, and the calculator 422 performs convolution operation of the mth convolution layer based on the mth input data to obtain the mth output data.

[0100] In response to determining that m = M, the data reader / writer 421 reads the (M-1)th output data as the Mth input data of the Mth convolution layer, and the calculator 422 performs convolution operation of the Mth convolution layer based on the Mth input data and outputs the Mth output data to the second generator 423. The Mth output data is a plurality of weight data.

[0101] For example, the data reader / writer 421 can include a data reader RDMA and a data writer WDMA. The data reader RDMA reads the first input data from the address storing the first input data in the storage unit and sends it to the calculator 422. The calculator 422 performs convolution operation of the first convolution layer on the first input data to obtain the first output data of the first convolution layer. The data writer WDMA writes the first output data into the address storing the second input data in the storage unit.

[0102] For example, the number of layers of the convolution operation currently performed by the calculator 422 can be counted, and the current layer number m can be determined by the count value. For example, M = 6, when m = < 6, after the calculator 422 completes the convolution operation of the current layer, the data writer WDMA writes the output data into the corresponding address in the storage unit, and the count of the number of layers of the convolution operation currently performed by the calculator 422 is incremented by 1, and then the data reader RDMA reads the input data of the next convolution layer from the storage unit to the calculator 422, and the calculator 422 performs the convolution operation until the count value m = 6. In response to m = 6, after completing the convolution operation of the 6th convolution layer, the calculator 422 outputs the 6th output data obtained by the convolution operation to the second generator 423.

[0103] For example, the 1st convolution layer input data is the luminance feature map F generated by the first generator 414 based on the luminance data Y, the Mth output data is a plurality of weight data Q, and the mth output data is the m+1th input data.

[0104] In the embodiment of the present disclosure, a calculation hardware (calculator 422) is instantiated, and the calculator 422 is multiplexed, so that the calculator 422 sequentially completes the operation of a plurality of convolution layers. In the calculation process, the input data and the output data of each convolution layer can also be multiplexed in the storage space. The maximum calculation data amount supported by the calculator 422 can be the calculation amount required by the 1st convolution layer C1. For example, the maximum size of the input data supported by the calculator 422 is 480x270, so the calculator 422 can also be used to perform the convolution operation of the 2nd convolution layer C2 to the 6th convolution layer C6. Multiplexing the calculator 422 can save calculation hardware resources and storage resources, and reduce the hardware structure.

[0105] FIG. 5 is a structural schematic diagram of an image processing apparatus according to another embodiment of the present disclosure.

[0106] As shown in FIG. 5, the image processing apparatus 500 includes a calculation module, a first cache unit 503, and a second cache unit 504. The calculation module includes a data reader and writer and a calculator 522, the data reader and writer includes a data reader RDMA and a data writer WDMA, and the calculator 522 includes N calculation units P1-PN and N activation units R1-RN.

[0107] In the embodiment of the present disclosure, the processing module writes the luminance feature map F into the first storage unit S1, for example, the first storage unit S1 can be a DDR. The data reader RDMA of the calculation module reads the mth input data required by the mth convolution operation in the neural network into the first cache unit 503, and the calculator 522 of the calculation module performs multi-path parallel convolution operation on the mth input data in the first cache unit by using a plurality of convolution kernels of the neural network.

[0108] In the embodiments of the present disclosure, the neural network includes M cascaded convolution layers, the first cache unit 503 includes N first storage spaces A1-A N, N is related to the number of input channels of the mth convolution layer in the M convolution layers of the neural network, 1

[0109] In the embodiments of the present disclosure, the capacity of the first storage spaces A1-A N is K*W, where K is related to the size of the convolution kernel, and W is related to the maximum data dimension in the M input data. For example, K can be the dimension of the convolution kernel, for example, the size of the convolution kernel is 3*3, K=3. For example, W can be the dimension of the input data of the first convolution layer C1, for example, the dimension of the input data of the first convolution layer C1 is 480*270, W=480.

[0110] The 16 first storage spaces A1-A16 can respectively cache three rows of feature data of the 16 input data. For example, for the convolution calculation process of the first convolution layer C1, the first input data is a 480*270 feature map. The data reader RDMA writes the first 3 rows (3*480*16bit, the brightness value of each pixel point is 16bit) of feature data in the 480*270 feature map to the first storage spaces A1-A16. After each channel completes the convolution operation on the first 3 rows*480 feature data, the data reader RDMA writes the 4th-6th rows of feature data (3*480*16bit) in the 480*270 feature map to the first storage spaces A1-A16. The data reader RDMA sequentially writes every 3 rows of feature data of the 480*270 feature map to the first storage spaces A1-A16, and the feature data written to each first storage space is the same. After the data cached in the first storage spaces A1-A16 is read by the calculation units P1-P16, the data cached in the first storage spaces A1-A16 is cleared to cache subsequent feature data.

[0111] When the calculator 522 performs convolution operation of multiple convolution layers, the N first storage spaces A1-A N can be multiplexed. Since the first storage spaces A1-A N can cache the data with the maximum dimension in the convolution operation, the first storage spaces A1-A N support caching the feature data participating in the operation each time in the convolution operation.

[0112] For the convolution calculation process of the second convolution layer C2, the second input data is a 240x135 feature map. The data reader RDMA writes the feature data of the first 3 rows (3x240x16 bits, the luminance value of each pixel point is 16 bits) in the 240x135 feature map into the first storage spaces A1-A16, and pads the spaces in the first storage spaces A1-A16 that are not written with data. For example, 0 is filled in each row of 240 spaces, so that 3x480 spaces in the first storage spaces A1-A16 are filled.

[0113] In the embodiment of the present disclosure, the second cache unit 504 includes N second storage spaces B1-BN. Each of the N second storage spaces B1-BN stores D convolution kernels. N is related to the number of input channels of the mth convolution layer in the neural network, D is related to the number of output channels of the mth convolution layer in the M convolution layers of the neural network, 1

[0114] For example, the second storage spaces B1-BN cache the convolution kernels required by the convolution operation currently performed by the second storage space B1-BN cache calculator 522. The storage spaces of the second storage spaces B1-BN support caching the convolution kernels required for participating in the convolution operation. For example, the amount of data that can be accommodated by each of the second storage spaces B1-BN is 16x9x16 bits (the size of the convolution kernel is 3x3, and each parameter of the convolution kernel is 16 bits). The convolution kernels can be pre-stored in the second storage unit S2. For FPGA, the convolution kernels can be pre-stored in the second storage unit S2 in a read-only memory (ROM) manner, or pre-loaded into the second storage unit S2 in a flash memory (FLASH) power-on initialization manner.

[0115] The amount of data that can be accommodated by the second storage unit S2 is 16x16x9x16 bits. The second storage unit S2 can accommodate a maximum of 16 groups of convolution kernels, each group including 16 convolution kernels. The number of groups of convolution kernels is the same as the number of input channels of each convolution layer, and the number of convolution kernels in each group is the same as the number of output channels of each convolution layer.

[0116] Since the calculation units P1-P16 are parallel calculation, in each clock cycle, the calculation units P1-P16 read 16x16 bit convolution kernel parameters (one convolution kernel parameter of the convolution kernel) from the corresponding second storage spaces B1-B16. In each clock cycle, the calculation units P1-P16 read 16x16x16 bit convolution kernel parameters from the second storage unit S2. Since the size of the convolution kernel is 3x3, the calculation units P1-P16 need 9 clock cycles to complete the convolution operation of one convolution layer.

[0117] In order to avoid that the bit width of the instantiated second storage unit S2 wastes a large amount of space, the second storage spaces B1-BN can be instantiated by using on-chip register resources.

[0118] In the embodiments of the present disclosure, the N calculation units P1-PN can realize N-channel data output. For example, the N calculation units P1-PN respectively perform multi-channel parallel convolution operation on the mth input data to obtain N convolution results of the mth convolution layer. When N=16, each calculation unit receives 16 feature maps and 16 convolution kernel parameters (one value in each convolution kernel) and performs 16 multiplication-addition operations in parallel. For a 3x3 convolution kernel, one convolution operation is completed every 9 clock cycles, and the convolution result of one feature map is output.

[0119] Each calculation unit can include a multiplier-adder and an adder. The multiplier-adder performs multiplication-addition operation of 16 feature maps and 16 convolution kernel parameters, and outputs one calculation result. In each clock cycle, the multiplication-addition operation includes 16 multiplications and 16 additions. For a 3x3 convolution kernel, the adder adds the calculation results output in 9 clock cycles to obtain the convolution result. In 9 clock cycles, the addition operation includes 9 additions.

[0120] In the embodiments of the present disclosure, the N activation units R1-RN can realize activation of N-channel data output. For example, the N activation units R1-RN respectively perform activation processing on the N convolution results to output the mth output data.

[0121] For example, the N activation units R1-RN respectively perform activation operation on the convolution results output by the N calculation units P1-PN. The N activation units R1-RN correspond to the N calculation units P1-PN.

[0122] For example, the activation unit can activate the convolution result based on a linear rectification function (Rectified Linear Unit, relu). The activation unit can perform piecewise function calculation on the convolution result output by the calculation unit according to formula (6):

[0123] Wherein, ft_in is the convolution result of the input activation unit, ft_out is the activated convolution result of the output of the activation unit, and k is the activation coefficient.

[0124] In the embodiments of the present disclosure, since 16 feature maps need to be calculated and 16 groups (each group has 16 convolution kernels) of convolution kernels need to be convolved in each convolution layer, 16 feature maps are output. In order to save resources, improve calculation efficiency, and reduce control logic complexity, the plurality of calculation units and the plurality of activation units in each convolution layer adopt parallel calculation. The plurality of calculation units and the plurality of activation units simultaneously perform parallel convolution on 16 feature maps and simultaneously output 16 feature maps.

[0125] In the embodiments of the present disclosure, when the convolution result output by the calculator 522 is not the convolution result of the last convolution layer, the data writer WDMA writes the convolution result to the specified address in the first storage unit S1 to participate in the convolution operation of the subsequent convolution layer. When the convolution result output by the calculator 522 is the convolution result of the last convolution layer, the data writer WDMA can write the weight data output by the last convolution layer to the second generator, so that the second generator generates the brightness enhancement data based on the weight data and the plurality of preset brightness lookup tables.

[0126] In the embodiments of the present disclosure, the calculator 522 realizes AI calculation on the brightness feature map through CNN to determine the image category of the display image, so as to adjust the brightness of the display image, reduce the average brightness of the display image, and also ensure the quality effect while reducing the display power consumption of the display module. In addition, the calculator 522, the first cache unit 503 and the second cache unit 504 are multiplexed in the multi-layer calculation process of the CNN, thereby saving hardware resources. The plurality of calculation units and the plurality of activation units in the calculator 522 are used for parallel calculation, thereby improving the calculation efficiency of each convolution operation.

[0127] FIG. 6 is a flowchart of an image processing method according to another embodiment of the present disclosure.

[0128] In the embodiments of the present disclosure, the image processing method 600 can include operation S610 to operation S640.

[0129] In operation S610, brightness data and color data of a display image are obtained.

[0130] In operation S620, the brightness data is convolved by using a neural network to obtain brightness enhancement data.

[0131] In operation S630, the brightness data is enhanced by using the brightness enhancement data to obtain target brightness data.

[0132] In operation S640, the target luminance data and the color data are converted into a target image, and the target image is used for outputting to a display module for display.

[0133] In the embodiments of the present disclosure, operations S610 to S640 are similar to the execution operations of the image processing apparatus 100 described above, and are not described again for simplicity.

[0134] In the embodiments of the present disclosure, the image processing method further includes: buffering the color data; and reading the color data when the target luminance data is determined, so that the target luminance data and the color data are converted into a target image.

[0135] In the embodiments of the present disclosure, in operation S620, the luminance data is convoluted by using the neural network to obtain the luminance enhancement data, including: partitioning the luminance data based on a data amount of input data supported by the neural network to obtain a plurality of partitions, the luminance data including luminance values of all pixel points in the display image; generating a luminance feature map by using luminance value statistical data of a plurality of pixel points in each partition of the plurality of partitions; and convoluting the luminance feature map by using the neural network to obtain the luminance enhancement data.

[0136] In the embodiments of the present disclosure, the luminance feature map is generated by using the luminance value statistical data of the plurality of pixel points in each partition of the plurality of partitions, including: determining a partition luminance value of each partition of the plurality of partitions based on a maximum luminance value and an average luminance value in the plurality of pixel points of each partition; and generating the luminance feature map according to the partition luminance value of each partition of the plurality of partitions.

[0137] In the embodiments of the present disclosure, in operation S620, the luminance data is convoluted by using the neural network to obtain the luminance enhancement data, including: convoluting the luminance data by using the neural network to obtain a plurality of weight data; and weighting and merging a plurality of preset luminance lookup tables by using the weight data to obtain the luminance enhancement data.

[0138] In the embodiment of the present disclosure, the neural network comprises M cascaded convolution layers; the image processing method 600 further comprises: reading the mth input data required by the mth convolution operation of the M convolution layers of the neural network to a first cache unit; and performing convolution operation on the mth input data in the first cache unit by using a plurality of convolution kernels of the neural network; wherein the first cache unit comprises N first storage spaces, N is related to the number of input channels of the mth convolution layer in the neural network, 1 < m ≤ M, and N and M are positive integers.

[0139] In the embodiment of the present disclosure, the convolution operation is performed on the mth input data to obtain the mth output data, comprising: performing multi-channel parallel convolution operation on the mth input data to obtain N convolution results; and performing activation processing on the N convolution results to obtain the mth output data.

[0140] In the embodiment of the present disclosure, the neural network comprises M cascaded convolution layers; the image processing method 600 further comprises: reading the mth input data required by the mth convolution operation of the M convolution layers of the neural network to a first cache unit; and performing convolution operation on the mth input data in the first cache unit by using a plurality of convolution kernels of the neural network; wherein the first cache unit comprises N first storage spaces, N is related to the number of input channels of the mth convolution layer in the neural network, 1 < m ≤ M, and N and M are positive integers.

[0141] In the embodiment of the present disclosure, the capacity of the first storage space is K*W, wherein K is related to the size of the convolution kernel, and W is related to the maximum data dimension in the M input data.

[0142] In the embodiment of the present disclosure, the neural network comprises M cascaded convolution layers; the image processing method 600 further comprises: reading the mth input data required by the mth convolution operation of the M convolution layers of the neural network to a first cache unit; and performing convolution operation on the mth input data in the first cache unit by using a plurality of convolution kernels of the neural network; wherein the first cache unit comprises N first storage spaces, N is related to the number of input channels of the mth convolution layer in the neural network, 1 < m ≤ M, and N and M are positive integers.

[0143] In the embodiment of the present disclosure, the display data is processed based on a first processing clock, and the convolution operation is performed based on a second processing clock, wherein the frequency of the second processing clock is greater than the frequency of the first processing clock.

[0144] FIG. 7 is a structural schematic diagram of a display device according to an embodiment of the present disclosure.

[0145] As shown in FIG. 7, the display device 700 includes a display module 701 and an image processing device 702.

[0146] In the embodiment of the present disclosure, the image processing device 702 outputs a target image based on a display image required to be displayed by the display module 701, and the display module 701 displays the target image.

[0147] In the embodiment of the present disclosure, the image processing device 702 is at least one of the image processing device 100, the image processing device 200, the image processing device 400 and the image processing device 500 described above, and will not be described again for the sake of simplicity.

[0148] FIG. 8 is a structural schematic diagram of a display device according to another embodiment of the present disclosure.

[0149] As shown in FIG. 8, the display device 800 includes a display module 801 and a processor 802.

[0150] In the embodiment of the present disclosure, the processor 802 executes the image processing method 600 described above for a display image required to be displayed by the display module 801, and outputs a target image. The display module 801 displays the target image.

[0151] In the embodiment of the present disclosure, the processor 802 can also execute the operations performed by at least one of the image processing device 100, the image processing device 200, the image processing device 400 and the image processing device 500, and will not be described again for the sake of simplicity.

[0152] It should be noted that in the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved in the technical solutions comply with the relevant legal regulations, necessary security measures are taken, and do not violate public order and good customs. In the technical solutions of the present disclosure, the authorization or consent of the user is obtained before the user's personal information is obtained or collected.

[0153] The computer program product of the first aspect can include one or more non-transitory computer-readable media storing instructions that, when executed, cause a processor of a computer-based system to perform any of the methods of the first aspect. The computer program product of the first aspect can include one or more non-transitory computer-readable media storing instructions that, when executed, cause a processor of a computer-based system to perform any of the methods of the first aspect.

[0154] Those skilled in the art will understand that the features recited in the various embodiments and / or claims of the present disclosure can be combined and / or interchanged, even if this is not explicitly stated in the present disclosure. In particular, the features recited in the various embodiments and / or claims of the present disclosure can be combined and / or interchanged, without departing from the spirit and teachings of the present disclosure. All such combinations and / or interchanges are within the scope of the present disclosure.

[0155] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although the above describes each embodiment separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. An image processing apparatus, comprising: The processing module is configured to acquire brightness and color data of the displayed image; as well as The calculation module is configured to perform convolution operations on the brightness data using a neural network to obtain brightness enhancement data; The processing module is further configured to enhance the brightness data using the brightness enhancement data to obtain target brightness data, and to convert the target brightness data and the color data into a target image, which is then output to the display module for display.

2. The apparatus according to claim 1, wherein, The processing module includes a first converter, a second converter, and a delay unit; The first converter is configured to convert the pixel values ​​of the displayed image into the brightness data and the color data; The delay unit is configured to delay the output of the color data to the second converter, so that the second converter receives the target brightness data and the color data simultaneously; The second converter is configured to convert the target brightness data and the color data into pixel values ​​of the target image.

3. The apparatus according to claim 2, wherein, The delay unit is also configured to: Write the color data into the storage unit; and The color data is read from the storage unit at the target time and output to the second converter; The target time is determined based on the time when the second converter receives the target brightness data.

4. The apparatus according to claim 1 or 2, wherein, The processing module includes: A first generator is configured to generate a luminance feature map based on the luminance data, and the luminance feature map is convolved by the computation module to obtain the luminance enhancement data; and An enhancer is configured to enhance the brightness data based on the brightness enhancement data to obtain the target brightness data.

5. The apparatus according to claim 4, wherein, The first generator is also configured as follows: Based on the amount of input data supported by the neural network, the brightness data is partitioned to obtain multiple partitions, and the brightness data includes the brightness values ​​of all pixels in the displayed image; as well as The brightness feature map is generated by using the brightness value statistics of multiple pixels in each of the multiple partitions.

6. The apparatus according to claim 5, wherein, The first generator is also configured as follows: The brightness value of each of the multiple partitions is determined based on the maximum brightness value and the average brightness value of each of the multiple pixels in each partition. as well as The brightness feature map is generated based on the brightness values ​​of each of the multiple partitions.

7. The apparatus according to claim 1, wherein, The computing module includes: A data reader / writer is configured to read the brightness data; A calculator is configured to perform convolution operations on the brightness data using the neural network to obtain multiple weight data; and The second generator is configured to use the weighted data to perform a weighted merging of multiple preset brightness lookup tables to obtain the brightness enhancement data.

8. The apparatus according to claim 7, wherein, The neural network comprises M cascaded convolutional layers; the data reader and the calculator perform the convolution operation of the m-th convolutional layer among the M convolutional layers in the following manner, where 1 ≤ m ≤ M, and m and M are positive integers: In response to determining m=1, the data reader reads the brightness data as the first input data of the first convolutional layer, and the calculator performs a convolution operation of the first convolutional layer based on the first input data to obtain the first output data; In response to determining 1 < m < M, the data reader reads the (m-1)th output data of the (m-1)th convolutional layer as the mth input data of the mth convolutional layer, and the calculator performs a convolution operation on the mth convolutional layer based on the mth input data to obtain the mth output data; as well as In response to determining m = M, the data reader reads the (M-1)th output data as the Mth input data of the Mth convolutional layer. The calculator performs a convolution operation on the Mth convolutional layer based on the Mth input data and outputs the Mth output data to the second generator. The Mth output data is the plurality of weight data.

9. The apparatus according to claim 8, wherein, The calculator includes: N computational units are configured to perform convolution operations on the m-th input data to obtain N convolution results; and N activation units are configured to activate the N convolution results to obtain the m-th output data.

10. The apparatus according to claim 1, further comprising a first cache unit; in, The neural network includes M cascaded convolutional layers; the computing module is further configured to read the m-th input data required for the convolution operation of the m-th layer in the M convolutional layers into the first cache unit, and the computing module uses multiple convolutional kernels of the neural network to perform convolution operation on the m-th input data in the first cache unit. The first cache unit includes N first storage spaces, where N is related to the number of input channels of the m-th convolutional layer in the neural network, 1 < m ≤ M, and N and M are positive integers.

11. The apparatus according to claim 10, wherein, The capacity of the first storage space is K*W, where K is related to the size of the convolution kernel and W is related to the maximum data dimension among the M input data.

12. The apparatus according to claim 1, further comprising a second buffer unit; in, The neural network includes M cascaded convolutional layers; the second cache unit includes N second storage spaces, each of which is configured to store D convolutional kernels; wherein N is related to the number of input channels of the m-th convolutional layer in the M convolutional layers, D is related to the number of output channels of the m-th convolutional layer in the neural network, 1 < m ≤ M, and D, N and M are positive integers.

13. The apparatus according to any one of claims 1-12, wherein, The processing module processes the display data based on a first processing clock, and the calculation module performs convolution operations based on a second processing clock, wherein the frequency of the second processing clock is greater than the frequency of the first processing clock.

14. An image processing method, comprising: Obtain the brightness and color data of the displayed image; as well as The brightness data is convolved using a neural network to obtain brightness-enhanced data; The brightness data is enhanced using the brightness enhancement data to obtain the target brightness data; as well as The target brightness data and the color data are converted into a target image, which is then output to the display module for display.

15. The method of claim 14, further comprising: Cache the color data; as well as When the target brightness data is determined, the color data is read, so that the target brightness data and the color data are converted into a target image.

16. The method of claim 14, wherein, The step of using a neural network to perform convolution operations on the brightness data to obtain brightness-enhanced data includes: Based on the amount of input data supported by the neural network, the brightness data is partitioned to obtain multiple partitions, and the brightness data includes the brightness values ​​of all pixels in the displayed image; The brightness feature map is generated using statistical data on the brightness values ​​of multiple pixels in each of the multiple partitions; and The brightness enhancement data is obtained by performing convolution operations on the brightness feature map using a neural network.

17. The method according to claim 16, wherein, The step of generating the brightness feature map by utilizing the brightness value statistics of multiple pixels in each of the multiple partitions includes: Based on the maximum and average brightness values ​​of multiple pixels in each of the multiple partitions, the brightness value of each partition is determined; and The brightness feature map is generated based on the brightness values ​​of each of the multiple partitions.

18. The method according to claim 14, wherein, The step of using a neural network to perform convolution operations on the brightness data to obtain brightness-enhanced data includes: The brightness data is convolved using the neural network to obtain multiple weight data; and The brightness enhancement data is obtained by weighting and merging multiple preset brightness lookup tables using the weighted data.

19. The method according to claim 18, wherein, The neural network comprises M cascaded convolutional layers; the convolution operation on the brightness data using the neural network yields multiple weight data, including: The convolution operation of the m-th convolutional layer in the M convolutional layers is performed as follows, where 1 ≤ m ≤ M, and m and M are positive integers: In response to determining m=1, the brightness data is read as the first input data of the first convolutional layer, and the convolution operation of the first convolutional layer is performed based on the first input data to obtain the first output data; In response to determining 1 < m < M, the (m-1)th output data of the (m-1)th convolutional layer is read as the m-th input data of the m-th convolutional layer, and a convolution operation of the m-th convolutional layer is performed based on the m-th input data to obtain the m-th output data; and In response to determining m = M, the (M-1)th output data is read as the Mth input data of the Mth convolutional layer, and the convolution operation of the Mth convolutional layer is performed based on the Mth input data to output the Mth output data. The output data of the Mth convolutional layer is the plurality of weight data.

20. The method according to claim 19, wherein, The step of performing a convolution operation on the m-th input data of the m-th convolutional layer to obtain the m-th output data includes: Perform a convolution operation on the m-th input data to obtain N convolution results; and Activation processing is performed on the N convolution results to obtain the m-th output data.

21. The method according to claim 14, wherein, The neural network comprises M cascaded convolutional layers; the method further comprises: The m-th input data required for the convolution operation of the m-th convolutional layer in the M convolutional layers is read into the first buffer unit; and The m-th input data in the first cache unit is convolved using multiple convolution kernels of the neural network. The first cache unit includes N first storage spaces, where N is related to the number of input channels of the m-th convolutional layer in the neural network, 1 < m ≤ M, and N and M are positive integers.

22. The method according to claim 21, wherein, The capacity of the first storage space is K*W, where K is related to the size of the convolution kernel and W is related to the maximum data dimension among the M input data.

23. The method according to claim 14, wherein, The neural network comprises M cascaded convolutional layers; the method further comprises: Read the multiple convolution kernels required for the convolution operation from the second cache unit; The second cache unit includes N second storage spaces, each of which is configured to store D convolutional kernels. N is related to the number of input channels of the m-th convolutional layer in the M convolutional layers, and D is related to the number of output channels of the m-th convolutional layer in the neural network. 1 < m ≤ M, and D, N, and M are positive integers.

24. The method according to any one of claims 14-23, wherein, Display data is processed based on a first processing clock, and convolution operations are performed based on a second processing clock, wherein the frequency of the second processing clock is greater than the frequency of the first processing clock.

25. A display device, comprising: Display module; as well as The image processing apparatus according to any one of claims 1-13 is configured to output a target image based on a display image to be displayed by the display module; The display module displays the target image.

26. A display device, comprising: A processor configured to perform the method of any one of claims 14-24; as well as The display module is configured to display the target image output by the processor.

Images