Image data processing method, storage medium, image processor and electronic device

Abstract

Embodiments of the present application disclose an image data processing method, a storage medium, an image processor and an electronic device. In the embodiments of the present application, a pixel row is received from an image sensor, wherein the pixel row is obtained by exposing the image sensor in a row interleaving manner according to a plurality of different exposure parameters; and the pixel row is fused based at least in part on a first movement between a first pixel and a second pixel, wherein the first pixel is a pixel in a pixel row corresponding to a first exposure parameter, the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter, and the first exposure parameter is different from the second exposure parameter. According to the embodiments of the present application, the pixel row is received from the image sensor in a row, and the fusion processing is performed based on the first movement between the pixels, so that the image fusion is performed without waiting for all the multiple frames of images to be output, thereby improving the image synthesis efficiency and meeting the requirement of the electronic device for high processing frame rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and in particular to an image data processing method, a storage medium, an image processor and an electronic device. BACKGROUND

[0002] With the continuous development of intelligent terminal technology, the use of electronic devices (such as smart phones, tablet computers, etc.) is becoming more and more popular. Most electronic devices are built-in image sensors, and with the increasing processing power of mobile terminals and the development of image sensor technology, users have higher and higher requirements for the quality of the captured images.

[0003] For example, more and more electronic devices now support high dynamic range image shooting. For example, some electronic devices use a DOLHDR (Digital Overlap High-Dynamic Range) scheme to shoot HDR images, but the current DOLHDR scheme has the problem of low image data processing frame rate, which is difficult to meet the requirements of electronic devices for high processing frame rate. SUMMARY

[0004] The embodiments of the present application provide an image data processing method, a storage medium, an image processor and an electronic device, which can meet the requirements of electronic devices for high processing frame rate.

[0005] In a first aspect, the embodiments of the present application provide an image data processing method, which comprises:

[0006] Step S1, receiving a pixel row from an image sensor, wherein the pixel row is obtained by the image sensor exposing in a row interleaving manner according to a plurality of different exposure parameters;

[0007] Step S2, performing fusion processing on the pixel row based at least in part on a first movement amount between a first pixel and a second pixel, wherein the first pixel is a pixel in a pixel row corresponding to a first exposure parameter, the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter, and the first exposure parameter is different from the second exposure parameter.

[0008] In a second aspect, the embodiments of the present application provide a storage medium having a computer program stored thereon, which, when executed on a computer, causes the computer to perform the image data processing method provided by any of the embodiments of the present application.

[0009] In a third aspect, the embodiments of the present application provide an image processor, which comprises:

[0010] receive, from the image sensor, a pixel row, wherein the pixel row is obtained by the image sensor exposing in a row-interleaved manner according to a plurality of different exposure parameters;

[0011] fuse the pixel row based at least in part on a first movement amount between a first pixel and a second pixel, wherein the first pixel is a pixel in a pixel row corresponding to a first exposure parameter, the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter, and the first exposure parameter is different from the second exposure parameter.

[0012] In a fourth aspect, an electronic device is provided, including:

[0013] an image sensor configured to perform exposure operations in a row-interleaved manner according to a plurality of different exposure parameters, and transmit obtained pixel rows to an image processor;

[0014] receive, from the image sensor, a pixel row; and

[0015] fuse the pixel row based at least in part on a first movement amount between a first pixel and a second pixel, wherein the first pixel is a pixel in a pixel row corresponding to a first exposure parameter, the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter, and the first exposure parameter is different from the second exposure parameter.

[0016] The scheme provided by the embodiments of the present application receives a pixel row from an image sensor, wherein the pixel row is obtained by the image sensor exposing in a row-interleaved manner according to a plurality of different exposure parameters; and fuses the pixel row based at least in part on a first movement amount between a first pixel and a second pixel, wherein the first pixel is a pixel in a pixel row corresponding to a first exposure parameter, the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter, and the first exposure parameter is different from the second exposure parameter. Through the scheme of the present application, a pixel row is received from the image sensor in rows, and fused based on the first movement amount between pixels, without waiting for multiple frames of images to be output before image fusion, thereby improving image synthesis efficiency, meeting the requirement of electronic devices for high processing frame rate, and also avoiding ghosting in the fused image. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0018] Figure 1 A first flowchart of an image data processing method provided by an embodiment of the present application.

[0019] Figure 2 A schematic diagram of pixel rows exposed and output in a row interleaving manner in an image data processing method provided by an embodiment of the present application.

[0020] Figure 3 A schematic diagram of pixel row rearrangement in an image data processing method provided by an embodiment of the present application.

[0021] Figure 4 A schematic diagram of a pixel unit row in an image data processing method provided by an embodiment of the present application.

[0022] Figure 5 A schematic diagram of a pixel unit movement amount calculation process in an image data processing method provided by an embodiment of the present application.

[0023] Figure 6 A schematic diagram of an image processor provided by an embodiment of the present application.

[0024] Figure 7 A first schematic diagram of an electronic device provided by an embodiment of the present application.

[0025] Figure 8 A second schematic diagram of an electronic device provided by an embodiment of the present application.

[0026] Figure 9 A third schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort belong to the protection scope of the present application.

[0028] In this document, reference to“an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase“in an embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments.

[0029] The embodiment of the present application provides an image data processing method, and the execution subject of the image data processing method can be an image processor provided by the embodiment of the present application or an electronic device integrated with the image processor, wherein the image processor can be realized in the form of hardware or software. The electronic device can be a smart phone, a tablet computer, a palm computer, a notebook computer or a desktop computer.

[0030] Please refer to Figure 1 , Figure 1 The first flowchart of the image data processing method provided by the embodiment of the present application is shown in FIG. 1. The specific flow of the image data processing method provided by the embodiment of the present application can be as follows:

[0031] In step S1, a pixel row is received from an image sensor, wherein the pixel row is obtained by exposing the image sensor in a row interleaving manner according to a plurality of different exposure parameters.

[0032] The image sensor in the embodiment of the present application adopts a rolling shutter, that is, the exposure operation is performed in a row-by-row exposure manner. When the HDR image is captured by using the rolling shutter, the HDR image can be generated by using the DOL (Digital Overlap) HDR. The DOL is also called line interleaving.

[0033] Taking two-exposure DOL as an example, wherein the two-exposure DOL refers to exposing the image sensor according to two different exposure parameters. For example, the plurality of different exposure parameters include a long exposure parameter and a medium exposure parameter. The image sensor generates two different exposure parameters, such as long exposure and medium exposure, and outputs images in turn after two exposures. Since the rolling shutter is exposed row by row, the exposure of the second row of pixel points is performed after the exposure of the first row of pixel points is completed, and so on, until the exposure of all pixel points is completed. Based on this, in order to reduce the exposure time interval between two images, the DOL technology is proposed, that is, the first row of medium exposure is exposed after the first row of long exposure is exposed. Since the time of medium exposure is shorter than that of long exposure, the first row of medium exposure pixel row is output earlier than the second row of long exposure pixel row.

[0034] Based on this, from the perspective of reading out image data, the long exposure image data and the short exposure image data are output in a row interleaving manner in turn. Please refer to Figure 2 , Figure 2The schematic diagram of the pixel row exposed and output in the row interleaving manner in the image data processing method provided by the embodiment of the present application is shown in FIG. 1. The output order of the image data is as follows: the first output pixel row is the first row L1 of the long-exposure image, the second output pixel row is the first row M1 of the medium-exposure image, the third output pixel row is the second row L2 of the long-exposure image, the fourth output pixel row is the second row M2 of the medium-exposure image, and so on, until all the pixel rows of the images corresponding to the two different exposure parameters are obtained.

[0035] In other embodiments, it can also be three-exposure DOL, which means that the image sensor is exposed according to three different exposure parameters. For example, the three different exposure parameters include a long-exposure parameter, a medium-exposure parameter and a short-exposure parameter. It can be understood that the number of exposure parameters can also be set to a value greater than 3.

[0036] In the related art, the high dynamic synthesis of the image is generally performed after the output of all the pixel rows of the images corresponding to two different exposure parameters is completed, which leads to the difficulty of the image synthesis speed in meeting the requirement of the electronic device on the processing frame rate.

[0037] To solve the problem, in the embodiment of the present application, the output pixel row is synthesized while the image sensor is exposed in the row interleaving manner.

[0038] Specifically, the image sensor exposes and outputs the pixel row to the image processor in the row interleaving manner.

[0039] The image processor receives the pixel row transmitted by the image sensor row by row.

[0040] In step S2, the pixel row is fused based at least in part on the first movement between the first pixel and the second pixel, wherein the first pixel is a pixel in the pixel row corresponding to the first exposure parameter, and the second pixel is a corresponding pixel in the pixel row corresponding to the second exposure parameter, and the first exposure parameter is different from the second exposure parameter.

[0041] Next, in order to more clearly explain the scheme of the embodiment of the present application, the three-exposure DOL is taken as an example for description. The image sensor outputs the pixel row exposed in the row interleaving manner according to three exposure parameters to the image processor row by row in the manner of long-exposure pixel row, medium-exposure pixel row and short-exposure pixel row alternately output.

[0042] It should be noted that the image data output by the image sensor is original RAW image data. The RAW image (i.e., Bayer image with a suffix of.raw) data is original data converted by a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) image sensor from a light source signal captured by the CMOS or the CCD image sensor into digital signal. The original RAW image data has a higher gray level and saves complete data information. The RGB three colors in the RAW image data can have multiple arrangement modes, such as RGGB, GRBG, GBRG, and BGGR. Taking the RGGB arrangement mode as an example, four adjacent single-color pixel points form a pixel unit. In a subsequent HDR synthesis process, the pixel unit is taken as a minimum unit for HDR synthesis processing.

[0043] Based on the above principle, for the image processor, at least two rows of pixel rows corresponding to each exposure parameter received by the image processor can be fused. Therefore, when the pixel rows received by the image processor meet the synthesis requirement, the received multiple pixel rows are fused to obtain pixel rows with a high dynamic range. Each time multiple pixel rows are received, the multiple pixel rows that have not been fused are fused. That is, steps S1 and S2 are executed in a pipeline manner, so that the row exposure operation and the fusion processing operation of the pixel row can be performed synchronously, without waiting for all the multiple frames of images to be output before image fusion is performed, thereby improving the image synthesis efficiency and meeting the requirement of the electronic device for a high processing frame rate.

[0044] In the formula, it is assumed that the number of exposure parameters is N, and when the number of pixel rows received by the image processor reaches k*N, it can be determined that the pixel rows received by the image processor meet the synthesis requirement. The value of k is set in advance according to the processing frame rate and the power consumption requirement, and satisfies the following condition: 2≤k≤K and k is an integer multiple of 2, and K is half of the height of the image output by the image sensor in one exposure operation.

[0045] For example, N=3 and k=2, that is, the image sensor adopts three-exposure DOL to output pixel rows, and the image processor can perform fusion processing of the pixel rows once every six pixel rows. The six pixel rows are subjected to HDR synthesis processing to obtain two pixel rows with a high dynamic range. In other embodiments, k can also be 4, 6, and the like.

[0046] In addition, for the image processor, the module for performing the image fusion operation can be started and the image fusion operation can be performed by setting a clock when the received pixel rows meet the synthesis requirement. After one synthesis operation is completed, the module can be controlled to enter a dormant state until the next time pixel rows meeting the synthesis requirement are received. In this way, the module is prevented from being in a working state all the time, and the power consumption of the image processor for image synthesis is reduced.

[0047] Next, the image fusion manner is described. The image processor receives pixel rows from the image sensor, and then performs fusion processing on the received pixel rows based at least in part on a first movement amount between a first pixel and a second pixel. The first pixel is a pixel in a pixel row corresponding to a first exposure parameter, and the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter. The corresponding pixel refers to a pixel obtained by exposure of the same pixel point on the image sensor, and the positions of the two pixels are the same.

[0048] The plurality of different exposure parameters at least include the first exposure parameter and the second exposure parameter, and the first exposure parameter is different from the second exposure parameter. A pixel in an exposure row corresponding to the first exposure parameter is referred to as a first pixel, and a pixel in an exposure row corresponding to the second exposure parameter is referred to as a second pixel. The movement amount between the first pixel and the second pixel reflects the displacement of the pixel in the exposure row corresponding to the first exposure parameter relative to the pixel in the pixel row corresponding to the second exposure parameter. Assuming that the exposure row corresponding to the first exposure parameter includes 100 pixels and the exposure row corresponding to the second exposure parameter includes 100 pixels, 100 first movement amounts between the 100 first pixels and their 100 corresponding second pixels need to be calculated. It can be understood that the movement amounts of pixels at different positions can be different.

[0049] Since the images corresponding to the two adjacent exposure parameters are continuously exposed, if the object in the shooting scene moves or the electronic device shakes, ghosting is likely to occur after HDR synthesis. To avoid this phenomenon, for two adjacent pixel unit rows, the first movement amount between the two pixel unit rows is calculated in units of pixel units, and at least part of the first movement amount is referred to for HDR synthesis processing in subsequent HDR synthesis.

[0050] In an embodiment, between step S1 and step S2, the method further includes: performing sorting processing on the plurality of pixel rows corresponding to the plurality of different exposure parameters, so that two pixel rows corresponding to the same exposure parameter are arranged adjacent to each other.

[0051] In this embodiment, the plurality of pixel rows that have not been fused are reordered each time the number of received pixel rows meets the synthesis requirement. Please refer to Figure 3, Figure 3 The image data processing method provided in the embodiment of the present application is shown in the following figure. Assuming that N=3 and k=2, that is, the image sensor outputs pixel rows in three-exposure DOL. When the image processor receives six pixel rows, the pixel rows can be fused once, as shown in the following figure. The received six pixel rows L1, M1, S2, L2, M2, and S2 are reordered so that two pixel rows corresponding to the same exposure parameter are arranged adjacently. The three pixel unit rows are fused to obtain a pixel row with high dynamic range. Figure 3

[0052] In an embodiment, the step S2 includes: taking two pixel rows corresponding to the same exposure parameter in the multiple pixel rows after the sorting as a pixel unit row to obtain multiple pixel unit rows; and performing fusion processing on the multiple pixel unit rows based at least in part on the first movement amount between the first pixel unit and the second pixel unit, wherein the multiple adjacent first pixels constitute the first pixel unit, and the multiple adjacent second pixels constitute the second pixel unit.

[0053] In the embodiment, after the sorting, two pixel rows corresponding to the same exposure parameter can constitute a pixel unit row, wherein the number of pixel units in the pixel unit row is half of the width of the pixel row. Taking two pixel rows corresponding to the same exposure parameter in the multiple pixel rows after the sorting as a pixel unit row can obtain multiple pixel unit rows, please refer to the following figure. Figure 4 Figure 4 The image data processing method provided in the embodiment of the present application is shown in the following figure. Assuming that N=3 and k=2, that is, the image sensor outputs pixel rows in three-exposure DOL. When the image processor receives six pixel rows, the pixel rows can be fused once, as shown in the following figure. The received six pixel rows L1, M1, S2, L2, M2, and S2 are reordered so that two pixel rows corresponding to the same exposure parameter are arranged adjacently. The three pixel unit rows are fused to obtain a pixel row with high dynamic range.

[0054] In the embodiment of the present application, the pixel row is composed of a row of single pixels. The pixel unit row is composed of a row of pixel units, and the pixel unit is described above.

[0055] In an embodiment, the fusion processing on the multiple pixel unit rows based at least in part on the first movement amount between the first pixel unit and the second pixel unit includes: calculating the first movement amount between the first pixel unit and the second pixel unit for each pair of two adjacent pixel unit rows; and performing fusion processing on the multiple pixel unit rows based at least in part on the first movement amount between the second pixel unit and the first pixel unit.

[0056] ​​In this embodiment, in order to improve the calculation efficiency, the movement is calculated in units of pixel units when calculating the first movement. Taking two exposures DOL as an example, the long exposure parameter is the first exposure parameter, the medium exposure parameter is the second exposure parameter, one long exposure pixel unit contains four first pixels, and one medium exposure pixel unit contains four second pixels. According to the pixel values of the four groups of first pixels and second pixels, the first movement between the long exposure pixel unit and the medium exposure pixel unit can be calculated.

[0057] In an embodiment, for each pair of adjacent two pixel unit rows, the first movement between the first pixel unit and the second pixel unit is calculated, including: for each pair of adjacent two pixel unit rows, the second movement of the second pixel unit and the first pixel unit in a plurality of color channels is calculated; and the first movement between the second pixel unit and the first pixel unit is determined according to the second movement in the plurality of color channels.

[0058] Please refer to Figure 5 , Figure 5 The image data processing method provided in the embodiment of the present application provides a schematic diagram of the movement calculation process of the pixel unit. The second movement of the second pixel and the first pixel in each color channel is calculated respectively, and the second movement corresponding to R, G and B channels is obtained respectively. Then, the first movement of the long exposure pixel unit and the medium exposure pixel unit is determined according to the values of the three second movements. For the R channel, the absolute value of the difference between the color value R1 of the R channel of the long exposure pixel unit and the color value R2 of the R channel of the medium exposure pixel unit is calculated, and the second movement corresponding to the R channel is obtained. The B channel can be calculated in the same way. Since one pixel unit has two G channel color values, the average value of the two G channel color values can be calculated first, and then the absolute value of the difference between the two average values is calculated to obtain the first movement corresponding to the G channel.

[0059] According to the above method, the first movement between the long exposure pixel unit row and the medium exposure pixel unit row can be calculated. According to the same method, the first movement between the medium exposure pixel unit row and the short exposure pixel unit row can be calculated. It can be understood that when the exposure parameter is greater than two, any two adjacent exposure parameters can be taken as the first exposure parameter and the second exposure parameter in step S2 to calculate the first movement between the pixels in the two adjacent pixel rows.

[0060] In addition, it should be noted that, since not all pixels in the pixel row are moved, after the first movement amount between the first pixel and the second pixel is calculated, the pixel row can be fused based at least in part on the first movement amount between the first pixel and the second pixel. For example, only the region where the pixel movement occurs is fused based on the first movement amount between the first pixel and the second pixel. Specifically, the moving region in the pixel row is identified first, and the pixel row is fused based on the first movement amount between the first pixel and the second pixel of the moving region.

[0061] Optionally, in an embodiment, the fusing of the plurality of pixel unit rows based at least in part on the first movement amount between the second pixel and the first pixel includes: determining exposure indexes corresponding to the plurality of pixel unit rows respectively; determining movement indexes corresponding to the plurality of pixel unit rows according to the first movement amount; and fusing the plurality of pixel unit rows based at least in part on the exposure indexes and the movement indexes to obtain a pixel row with a high dynamic range.

[0062] After the first movement amount is obtained, the exposure index and the movement index used as the HDR synthesis parameter are determined. The exposure index includes a target exposure parameter corresponding to each pixel unit, i.e., for each pixel unit, the pixel unit value corresponding to the target exposure parameter is selected as the reference pixel value during synthesis. The movement index includes movement information corresponding to each pixel unit, i.e., for each pixel unit, the corresponding movement information indicates whether the pixel unit is moved.

[0063] For the three-exposure DOL, two first movement amounts are calculated, i.e., the first movement amount between the long-exposure pixel unit row and the medium-exposure pixel unit row, and the first movement amount between the medium-exposure pixel unit row and the short-exposure pixel unit row.

[0064] It can be understood that, assuming that the lengths of the long-exposure pixel unit row and the medium-exposure pixel unit row are 10 (i.e., the long-exposure pixel unit row and the medium-exposure pixel unit row each include 10 pixel units), the first movement amount between the long-exposure pixel unit row and the medium-exposure pixel unit row includes 10 values, which are the first movement amounts between the 10 pixel unit pairs respectively.

[0065] During HDR synthesis, for a pixel unit, a pixel unit pair with a smaller first movement amount is selected from the pixel unit pair formed by the long-exposure pixel unit and the medium-exposure pixel unit, and the pixel unit pair formed by the medium-exposure pixel unit and the short-exposure pixel unit for synthesis processing.

[0066] If the first movement amount of the pixel unit pair composed of the medium exposure pixel unit and the short exposure pixel unit is less than the first movement amount of the pixel unit pair composed of the long exposure pixel unit and the medium exposure pixel unit, in order to avoid ghosting, the pixel unit pair composed of the medium exposure pixel unit and the short exposure pixel unit is selected for the synthesis processing. Then, the pixel unit value with the smaller exposure parameter in the pixel unit pair is taken as the reference pixel value. That is, in the exposure index, the exposure parameter corresponding to the pixel unit is short exposure, and in the HDR synthesis, the pixel unit value corresponding to the short exposure parameter is selected as the reference pixel value, and the pixel unit value corresponding to the medium exposure parameter is used to compensate the reference pixel value.

[0067] Conversely, if the first movement amount of the pixel unit pair composed of the medium exposure pixel unit and the short exposure pixel unit is greater than the first movement amount of the pixel unit pair composed of the long exposure pixel unit and the medium exposure pixel unit, in order to avoid ghosting, the pixel unit pair composed of the long exposure pixel unit and the medium exposure pixel unit is selected for the synthesis processing. In the exposure index, the exposure parameter corresponding to the pixel unit pair is the medium exposure parameter, and in the HDR synthesis, the pixel unit value corresponding to the medium exposure parameter is selected as the reference pixel value, and the pixel unit value corresponding to the long exposure parameter is used to compensate the reference pixel value. Wherein, in the HDR synthesis, the weight corresponding to each pixel unit value used for synthesis can be calculated according to the respective pixel unit value.

[0068] It can be understood that for the two-exposure DOL, the pixel unit value with the smaller exposure parameter can be directly selected as the reference pixel value. Then, the HDR synthesis is performed according to the movement index determined according to the first movement amount.

[0069] In addition, regarding the movement index, the movement index indicates whether movement occurs at the pixel unit. Wherein, when the first movement amount is greater than a preset threshold value, it is determined that movement occurs at the pixel unit, and the pixel unit is marked as 1 in the movement index, indicating that movement occurs, otherwise, it is marked as 0, indicating that no movement occurs. For the pixel unit without movement, in the HDR synthesis, multiple pixel unit values can be fused, and for the pixel unit with movement, only one pixel unit value is selected to avoid ghosting in the HDR image obtained by synthesis. Wherein, the value of the preset threshold value can be an empirical value.

[0070] According to the above manner, for the three pixel unit rows in Figure 4 , the synthesis objects used by different pixel units are different in synthesis. For example, the long exposure pixel unit value and the medium exposure pixel unit value can be used for synthesis at the first pixel unit, and only the short exposure pixel unit value is used at the fifth pixel unit. Through such a synthesis manner, two rows of pixel rows with high dynamic range are finally obtained.

[0071] The image processor obtains an output image with high dynamic range according to the obtained multiple rows of pixel rows with high dynamic range. The image processor can send the output image to the application processor for further processing, or output the output image to a display screen for display.

[0072] Alternatively, in another embodiment, the image processor outputs the obtained two rows of pixel rows with high dynamic range to the application processor row by row, and the application processor obtains an output image with high dynamic range according to the received multiple rows of pixel rows with high dynamic range, and performs further processing or outputs the output image to a display screen for display.

[0073] In the implementation, the order of execution of the steps described in the application is not limited, and some steps can be performed in other orders or simultaneously without conflict.

[0074] As can be seen from the above, the image data processing method provided in the embodiments of the application receives a pixel row from an image sensor, wherein the pixel row is obtained by the image sensor in a row interleaving manner according to multiple different exposure parameters; and performs fusion processing on the pixel row based at least in part on a first movement amount between a first pixel and a second pixel, wherein the first pixel is a pixel in a pixel row corresponding to a first exposure parameter, the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter, and the first exposure parameter is different from the second exposure parameter. Through the scheme of the application, the pixel row is received from the image sensor row by row, and fusion processing is performed based on the first movement amount between the pixels, so that image fusion is performed without waiting for multiple images to be output, the image synthesis efficiency is improved, the requirement of the electronic device for high processing frame rate can be met, and ghosting in the fused image can be avoided.

[0075] In an embodiment, an image processor is also provided. Please refer to Figure 6 , Figure 6 FIG. 3 is a structural schematic diagram of an image processor 300 provided in the embodiments of the application. The image processor 300 is applied to an electronic device, and the image processor 300 includes a data receiving module 301 and an image fusion module 302, as follows:

[0076] The data receiving module is configured to receive a pixel row from an image sensor, wherein the pixel row is obtained by the image sensor in a row interleaving manner according to multiple different exposure parameters;

[0077] The image fusion module is configured to fuse the pixel rows based at least in part on a first movement amount between a first pixel and a second pixel, wherein the first pixel is a pixel in a pixel row corresponding to a first exposure parameter, and the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter, and the first exposure parameter is different from the second exposure parameter.

[0078] In some embodiments, the image fusion module 302 is further configured to sort the pixel rows corresponding to the plurality of different exposure parameters, such that two pixel rows corresponding to a same exposure parameter are arranged adjacently.

[0079] In some embodiments, the image fusion module 302 is further configured to arrange two pixel rows corresponding to a same exposure parameter as a pixel cell row after the sorting, to obtain a plurality of pixel cell rows.

[0080] The image fusion module is configured to fuse the pixel rows based at least in part on a first movement amount between a first pixel and a second pixel, wherein the first pixel is a pixel in a pixel row corresponding to a first exposure parameter, and the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter, and the first exposure parameter is different from the second exposure parameter.

[0081] In some embodiments, the image fusion module 302 is further configured to calculate, for each pair of two adjacent pixel cell rows, a first movement amount between a first pixel cell and a second pixel cell.

[0082] The image fusion module is configured to fuse the pixel rows based at least in part on a first movement amount between a first pixel and a second pixel, wherein the first pixel is a pixel in a pixel row corresponding to a first exposure parameter, and the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter, and the first exposure parameter is different from the second exposure parameter.

[0083] In some embodiments, the image fusion module 302 is further configured to calculate, for each pair of two adjacent pixel cell rows, a second movement amount between the second pixel cell and the first pixel cell in a plurality of color channels.

[0084] The image fusion module is configured to determine the first movement amount between the second pixel cell and the first pixel cell according to the second movement amount in the plurality of color channels.

[0085] In some embodiments, the image fusion module 302 is further configured to determine an exposure index corresponding to each of the plurality of pixel cell rows.

[0086] The image fusion module is configured to determine a movement index corresponding to the plurality of pixel cell rows according to the first movement amount.

[0087] The image fusion module is configured to fuse the plurality of pixel cell rows based at least in part on the exposure index and the movement index, to obtain a pixel row with a high dynamic range.

[0088] In some embodiments, the image processor 300 further comprises:

[0089] a data sending module, configured to output the pixel rows with high dynamic range to an application processor row by row.

[0090] In some embodiments, the plurality of different exposure parameters comprises a long exposure parameter, a medium exposure parameter and a short exposure parameter.

[0091] Alternatively, the plurality of different exposure parameters comprises a long exposure parameter and a medium exposure parameter.

[0092] It should be noted that the image processor provided by the embodiments of the present application and the image data processing method in the above embodiments belong to the same concept. Any method provided in the image data processing method embodiment can be implemented through the image processor. For details, refer to the image data processing method embodiment, which will not be described here.

[0093] As can be seen from the above, the image processor provided by the embodiments of the present application receives pixel rows from an image sensor, wherein the pixel rows are obtained by the image sensor in a row interleaving manner according to a plurality of different exposure parameters; and performs fusion processing on the pixel rows based at least in part on a first movement amount between a first pixel and a second pixel, wherein the first pixel is a pixel in a pixel row corresponding to a first exposure parameter, the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter, and the first exposure parameter is different from the second exposure parameter. Through the scheme of the present application, the pixel rows are received from the image sensor in rows, and the fusion processing is performed based on the first movement amount between the pixels. Without waiting for all the images to be output before image fusion, the image synthesis efficiency is improved, which can meet the requirement of the electronic device for high processing frame rate, and can also avoid the occurrence of ghosting in the fused image.

[0094] The embodiments of the present application also provide an electronic device. The electronic device can be a smart phone, a tablet computer or the like. Please refer to Figure 7 , Figure 7 The first structure diagram of the electronic device provided by the embodiments of the present application is shown. The electronic device 400 comprises an image sensor 401 and an image processor 300, wherein the image sensor 401 is configured to perform an exposure operation in a row interleaving manner according to a plurality of different exposure parameters, and transmit the obtained pixel rows to the image processor; the image processor 300 is configured to receive the pixel rows from the image sensor; and perform fusion processing on the pixel rows based at least in part on a first movement amount between a first pixel and a second pixel, wherein the first pixel is a pixel in a pixel row corresponding to a first exposure parameter, the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter, and the first exposure parameter is different from the second exposure parameter.

[0095] The image sensor 401 can include a color filter array (e.g., a Bayer filter). The image sensor can acquire light intensity and wavelength information captured with each imaging pixel of the image sensor and provide a set of raw image data that can be processed by an image signal processor.

[0096] The image processor 300 can be an image signal processor (ISP processor) that can process the raw image data pixel by pixel in multiple formats. For example, each image pixel can have a bit depth of 8, 10, 12, or 14 bits, and the image signal processor can perform one or more image data processing operations on the raw image data, collect statistical information about the image data, etc. The image data processing operations can be performed with the same or different bit depth precisions. The raw image data can be stored in an image memory after being processed by the image signal processor. The image signal processor can also receive image data from the image memory.

[0097] The image signal processor can also receive processing data from the image memory and perform image data processing on the processing data in the raw domain and in RGB and YCbCr color spaces. The processed image data can be output to an application processor for viewing by a user and / or further processing by a graphics engine or GPU (Graphics Processing Unit). In addition, the output of the image signal processor can be sent to the image memory, and a display can read the image data from the image memory. In an embodiment, the image memory can be configured to implement one or more frame buffers.

[0098] The image captured by the camera can be transmitted to the image signal processor for processing. After the image signal processor processes the image, it can send statistical data of the image (e.g., brightness of the image, contrast value of the image, color of the image, etc.) to the control logic. The control logic can determine control parameters of the camera according to the statistical data, so that the camera can perform automatic focusing, automatic exposure, etc. according to the control parameters. The image can be stored in an image memory after being processed by the image signal processor. The image signal processor can also read the image stored in the image memory for processing. In addition, the image can be directly sent to a display for display after being processed by the image signal processor. The display can also read the image in the image memory for display.

[0099] In some embodiments, please refer to Figure 8 , Figure 8 A second structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0100] The electronic device 400 also includes an application processor 402, which receives pixel rows with high dynamic range sent by the image processor 300 and obtains an output image with high dynamic range based on a plurality of the pixel rows with high dynamic range.

[0101] In some embodiments, please refer to Figure 9 , Figure 9 This is a third structural diagram of the electronic device provided in the embodiments of this application. The electronic device 400 further includes: a radio frequency circuit 403, a display screen 404, a control circuit 405, an input unit 406, an audio circuit 407, a sensor 408, and a power supply 409. The processor 401 is electrically connected to the radio frequency circuit 403, the display screen 404, the control circuit 405, the input unit 406, the audio circuit 407, the sensor 408, and the power supply 409.

[0102] The radio frequency circuit 403 is used to transmit and receive radio frequency signals to communicate with network devices or other electronic devices via wireless communication.

[0103] The display screen 404 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic devices, which can be composed of images, text, icons, videos, and any combination thereof.

[0104] The control circuit 405 is electrically connected to the display screen 404 and is used to control the display screen 404 to display information.

[0105] The input unit 406 can be used to receive input numeric or character information or user characteristic information (such as fingerprints), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. The input unit 406 may include a fingerprint recognition module.

[0106] The audio circuit 407 provides an audio interface between the user and the electronic device via a speaker and a microphone. The audio circuit 407 includes a microphone, which is electrically connected to the processor 401. The microphone is used to receive voice information input by the user.

[0107] Sensor 408 is used to collect information about the external environment. Sensor 408 may include one or more sensors such as an ambient light sensor, an accelerometer, and a gyroscope.

[0108] The power supply 409 is used to supply power to the various components of the electronic device 400. In some embodiments, the power supply 409 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0109] Although not shown in the figure, the electronic device 400 can further include a Bluetooth module and the like, which will not be described herein.

[0110] As can be seen from the above, the embodiment of the present application provides an electronic device, an image processor of the electronic device, receiving a pixel row from an image sensor, wherein the pixel row is obtained by the image sensor exposing in a row interleaving manner according to a plurality of different exposure parameters; performing fusion processing on the pixel row based at least in part on a first movement amount between a first pixel and a second pixel, wherein the first pixel is a pixel in a pixel row corresponding to a first exposure parameter, the second pixel is a corresponding pixel in a pixel row corresponding to a second exposure parameter, and the first exposure parameter is different from the second exposure parameter. Through the scheme of the present application, the pixel row is received from the image sensor in rows, and the fusion processing is performed based on the first movement amount between the pixels, without waiting for all the output of the multiple frames of images before performing image fusion, thereby improving the image synthesis efficiency, meeting the requirement of the electronic device for high processing frame rate, and also avoiding the occurrence of ghosting in the fused image.

[0111] The embodiment of the present application further provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and when the computer program runs on a computer, the computer executes the image data processing method in any of the above embodiments.

[0112] It should be noted that all or part of the steps of the various methods of the above embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer readable storage medium, which can include but is not limited to a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0113] In addition, the terms "first", "second" and "third" and the like in the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but some embodiments also include steps or modules not listed, or some embodiments also include other steps or modules inherent to the process, method, product or device.

[0114] The image data processing method, the storage medium, the image processor and the electronic device provided by the embodiments of the present application are described in detail above. The principles and implementation manners of the present application are described by applying specific examples in this paper, and the above embodiment description is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. An image data processing method, comprising: Step S1: Receive pixel rows from the image sensor, wherein the pixel rows are obtained by the image sensor through row interleaving exposure according to multiple different exposure parameters; sort the multiple pixel rows corresponding to the multiple different exposure parameters so that two pixel rows corresponding to the same exposure parameter are arranged adjacently, and the image data output by the image sensor is RAW image data. Step S2: Take two pixel rows with the same exposure parameter in the sorted pixel rows as a pixel unit row to obtain multiple pixel unit rows; for each pair of adjacent pixel unit rows, calculate the second movement amount of the second pixel unit and the first pixel unit in multiple color channels; determine the first movement amount between the second pixel unit and the first pixel unit based on the second movement amount in the multiple color channels; perform fusion processing on the multiple pixel unit rows at least in part based on the first movement amount between the first pixel unit and the second pixel unit, wherein the first pixel is a pixel in the pixel row corresponding to the first exposure parameter, the second pixel is a corresponding pixel in the pixel row corresponding to the second exposure parameter, the first exposure parameter is different from the second exposure parameter, multiple adjacent first pixels constitute the first pixel unit, and multiple adjacent second pixels constitute the second pixel unit.

2. The image data processing method according to claim 1, characterized in that, Steps S1 and S2 are executed in an assembly line manner.

3. The image data processing method as described in claim 1, characterized in that, The fusion processing of the plurality of pixel unit rows, based at least in part on a first movement amount between the second pixel and the first pixel, includes: Determine the exposure index corresponding to each of the plurality of pixel unit rows; The movement index corresponding to the plurality of pixel unit rows is determined based on the first movement amount; The plurality of pixel unit rows are fused based at least in part on the exposure index and the movement index to obtain pixel rows with high dynamic range.

4. The image data processing method as described in claim 3, characterized in that, After obtaining the pixel row with high dynamic range, the method further includes: The pixel rows with high dynamic range are output line by line to the application processor.

5. The image data processing method according to any one of claims 1 to 4, characterized in that, The various exposure parameters include long exposure parameters, medium exposure parameters, and short exposure parameters; Alternatively, the multiple different exposure parameters may include long exposure parameters and medium exposure parameters.

6. A storage medium having a computer program stored thereon, characterized in that, When the computer program is run on a computer, it causes the computer to perform the image data processing method as described in any one of claims 1 to 5.

7. An image processor, characterized in that, include: The data receiving module is used to receive pixel rows from the image sensor, wherein the pixel rows are obtained by the image sensor exposing the image sensor in a row-interleaved manner according to multiple different exposure parameters; the multiple pixel rows corresponding to the multiple different exposure parameters are sorted so that two pixel rows corresponding to the same exposure parameter are arranged adjacently, and the image data output by the image sensor is RAW image data; An image fusion module is used to group two pixel rows corresponding to the same exposure parameter in the sorted pixel rows into a single pixel unit row, resulting in multiple pixel unit rows. For each pair of adjacent pixel unit rows, a second movement amount between the second pixel unit and the first pixel unit in multiple color channels is calculated. Based on the second movement amount in the multiple color channels, a first movement amount between the second pixel unit and the first pixel unit is determined. The multiple pixel unit rows are fused based at least in part on the first movement amount between the first pixel unit and the second pixel unit, wherein the first pixel is a pixel in the pixel row corresponding to the first exposure parameter, the second pixel is a corresponding pixel in the pixel row corresponding to the second exposure parameter, the first exposure parameter is different from the second exposure parameter, multiple adjacent first pixels constitute the first pixel unit, and multiple adjacent second pixels constitute the second pixel unit.

8. The image processor as described in claim 7, characterized in that, Also includes: The data transmission module is used to output the pixel rows with high dynamic range obtained by the fusion processing to the application processor line by line.

9. An electronic device, characterized in that, include: An image sensor is used to perform an exposure operation in a row-interleaved manner according to multiple different exposure parameters, and transmit the resulting pixel rows to an image processor. The image data output by the image sensor is RAW image data. The image processor is used to receive pixel rows from the image sensor; The pixel rows corresponding to the multiple different exposure parameters are sorted so that two pixel rows corresponding to the same exposure parameter are arranged adjacently. as well as Two pixel rows with the same exposure parameter in the sorted pixel rows are combined into a single pixel unit row to obtain multiple pixel unit rows. For each pair of adjacent pixel unit rows, the second movement amount between the second pixel unit and the first pixel unit in multiple color channels is calculated. Based on the second movement amount in the multiple color channels, the first movement amount between the second pixel unit and the first pixel unit is determined. The plurality of pixel unit rows are fused based at least in part on a first movement amount between a first pixel unit and a second pixel unit, wherein the first pixel is a pixel in the pixel row corresponding to a first exposure parameter, the second pixel is a corresponding pixel in the pixel row corresponding to a second exposure parameter, the first exposure parameter is different from the second exposure parameter, a plurality of adjacent first pixels constitute a first pixel unit, and a plurality of adjacent second pixels constitute a second pixel unit.

10. The electronic device as claimed in claim 9, characterized in that, It also includes the application processor; The image processor is further configured to output the pixel rows with high dynamic range obtained by the fusion process to the application processor line by line; The application processor is configured to obtain an output image with high dynamic range based on a plurality of the pixel rows having high dynamic range.

Images