High dynamic range image processing method, system and medium based on multi-line sensor

Abstract

The present invention discloses a high dynamic range image processing method, system and medium based on a multi-line sensor, the method comprising: dividing the pixel rows of each RGB component in the color multi-line sensor so that each RGB component forms at least two groups of photosensitive units; completing the superposition of the scan data in all photosensitive units in each RGB component to form TDI data corresponding to each group of photosensitive units; collecting each group of TDI data, calculating the corresponding pixel value adjustment coefficient; setting the exposure reference time, calculating the pixel adjustment value of all pixel positions in each group of TDI data; obtaining row image data with a high dynamic range. Based on the characteristics of the multi-line sensor, the present invention can continuously image the target object multiple times on different pixel rows and combine with a flexible TDI superposition method to realize the HDR function, the output frame rate / line frequency is not affected, the time-space correlation is very high, and the time-space consistency of the synthesized image is good.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision, and in particular relates to a high dynamic range image processing method, system and medium based on a multi-line sensor. Background Art

[0002] The increasing requirements for detection accuracy and efficiency in the field of machine vision have led to the increasing application of time delayed integration (TDI) cameras. Compared with ordinary cameras that generally have only one row of photosensitive pixels, the typical characteristics of TDI cameras are high line frequency and multiple lines of photosensitive lines inside. For example, TDI cameras with 4 rows of photosensitive pixels are already relatively common, and there are more TDI cameras with 12 or even 16 rows of photosensitive pixels. Compared with higher-order TDI cameras such as 256-order (rows), the characteristics of these 4-16 photosensitive pixel rows TDI cameras are that the sensor itself usually does not have TDI function. Generally, all rows need to be read from the sensor and the TDI operation is completed in the back-end processor such as FPGA, which makes it possible to realize more functions such as HDR function.

[0003] Patent document CN117082366A discloses a linear array TDI image processing method, device and storage medium, which divides the linear array TDI sensor into multiple sensor arrays with different exposure times, and improves the image contrast by shooting linear array images with different exposure times such as low, medium and high, and performing image fusion. At the same time, the linear array TDI technology can further increase the sensitivity of the camera, which is suitable for high-speed motion and high-line frequency visual detection scenes. However, on the one hand, this method can only be applied to black and white cameras, and cannot be effectively used for the RGB component distribution of color cameras. On the other hand, due to the existence of multiple sensor arrays with different exposure times, the overall brightness balance is affected.

[0004] Therefore, in order to solve the above problems, the present invention provides a high dynamic range image processing method, system and medium based on a multi-line sensor. Summary of the invention

[0005] The purpose of the present invention is to overcome the above problems existing in the prior art and to provide a high dynamic range image processing method, system and medium based on a multi-line sensor.

[0006] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0007] The high dynamic range image processing method based on multi-line sensor includes:

[0008] Dividing the pixel rows of each RGB component in the color multi-line sensor so that each RGB component forms at least two groups of photosensitive units, each group of photosensitive units includes a plurality of consecutive pixel rows, and the exposure time of each pixel row is independently adjusted;

[0009] Obtain the scan data of each pixel row, complete the superposition of the scan data in all photosensitive units in each RGB component, and form the TDI data corresponding to each group of photosensitive units;

[0010] Collect pixel values ​​of all pixel positions in each group of TDI data and calculate corresponding pixel value adjustment coefficients;

[0011] Obtaining the exposure time corresponding to each group of TDI data, setting the exposure reference time, and calculating the corresponding pixel adjustment value according to the ratio of the exposure reference time to the exposure time of all pixel positions in each group of TDI data, so as to adjust the pixel values ​​of all pixel positions in each group of TDI data to the same energy level;

[0012] The pixel value adjustment coefficient and the pixel adjustment value of the same pixel position of all TDI data in each RGB component are obtained, and the first fused pixel value of the same pixel position is obtained by weighted calculation, so as to obtain row image data with a high dynamic range.

[0013] Furthermore, the scan data superposition is processed by summing or averaging the scan data of each pixel row in each group of photosensitive units.

[0014] Furthermore, the closer the pixel values ​​corresponding to all pixel positions in each group of TDI data are to half of the maximum pixel value, the larger the pixel value adjustment coefficient is, and vice versa.

[0015] Furthermore, adjusting the pixel values ​​of all pixel positions in each group of TDI data to a uniform energy level includes: setting an exposure reference time, and calculating a corresponding pixel adjustment value based on a ratio of the exposure reference time to the exposure time of all pixel positions in each group of TDI data.

[0016] Furthermore, image normalization processing is performed before collecting pixel values ​​of all pixel positions in each group of TDI data.

[0017] Furthermore, the multi-line sensor is a color multi-line sensor, a TDI coefficient of the color multi-line sensor is greater than or equal to two, and the number of the color multi-line sensor is one.

[0018] Furthermore, when the number of pixel rows in the photosensitive unit is greater than or equal to two, the method further includes:

[0019] Acquire the first data and the second data output by each group of photosensitive units, and store the first data row by row in the cache space with different addresses; wherein the first data represents the image data generated by the first row of pixels in the photosensitive unit collecting different positions of the object to be tested; and the second data represents the image data generated by the non-first row of pixels in the photosensitive unit collecting different positions of the object to be tested;

[0020] The stored data to be superimposed is read, superimposed with the second data at the same position, and the data to be superimposed for the next superimposition process is formed, and is stored again in the original cache space of the data to be superimposed; the superimposition process is repeated until the first data at the same position is superimposed with all the second data, and the image data after the superposition is completed is output; wherein the data to be superimposed includes the first data, and the image data after the first data and the second data are superimposed.

[0021] Further, at least two color multi-line sensors are installed, each color multi-line sensor has a TDI coefficient of one, and the incident light is divided into multiple paths and projected to each color multi-line sensor with equal optical path, and each color multi-line sensor is delayed by one pixel position in turn along the moving direction of the object to be measured; wherein the pixel interval of the color multi-line sensor is a pixel width, and the number of the color multi-line sensors is N, N=a+1;

[0022] The pixel rows of the same component in each color multi-line sensor are summarized in sequence and divided into at least two groups of photosensitive units to calculate the first fused pixel values ​​of different pixel positions in each RGB component, thereby obtaining and outputting RGB row image data with a high dynamic range.

[0023] Furthermore, it also includes:

[0024] Extracting image information collected by each color multi-line sensor and comparing them to obtain a pixel deviation value, wherein the exposure time of each color multi-line sensor is the same;

[0025] Obtain pixel deviation values ​​of each color multi-line sensor and identify the color multi-line sensor to be corrected;

[0026] The weighted average algorithm is used to perform pixel correction on the corrected color multi-line sensor.

[0027] The present invention also provides a high dynamic range image processing system based on a multi-line sensor, comprising:

[0028] A parameter adjustment module, used for dividing the pixel rows of each RGB component in the color multi-line sensor so that each RGB component forms at least two groups of photosensitive units, each group of photosensitive units includes a number of consecutive pixel rows, and the exposure time of each pixel row is adjusted independently;

[0029] The data superposition module is used to obtain the scan data of each pixel row, complete the scan data superposition of all photosensitive units in each RGB component, and form the TDI data corresponding to each group of photosensitive units;

[0030] A coefficient calculation module is used to collect pixel values ​​of all pixel positions in each group of TDI data and calculate the corresponding pixel value adjustment coefficient;

[0031] An energy unification module is used to obtain the exposure time corresponding to each group of TDI data, set the exposure reference time, and calculate the corresponding pixel adjustment value according to the ratio of the exposure reference time to the exposure time of all pixel positions in each group of TDI data, so as to adjust the pixel values ​​of all pixel positions in each group of TDI data to the same energy level;

[0032] The weighted calculation module is used to obtain the pixel value adjustment coefficient and pixel adjustment value of the same pixel position of all TDI data in each RGB component, and obtain the first fused pixel value of the same pixel position by weighted calculation, so as to obtain row image data with high dynamic range.

[0033] The present invention also provides a computer-readable storage medium, comprising a computer program, wherein the computer program implements the above processing method when executed by a processor.

[0034] The beneficial effects of the present invention are:

[0035] (1) The present invention can perform high dynamic range image processing for color multi-line sensors. By dividing each RGB component into at least two groups of photosensitive units and calculating TDI data, fused data of different pixel positions in each RGB component can be obtained; and the pixel values ​​of all pixel positions in each group of TDI data are adjusted to the same energy level in combination with the exposure time. A smaller exposure reference time can also be set to prevent value overflow.

[0036] (2) The present invention calculates the pixel values ​​of all pixel positions in each group of TDI data, and can adjust the pixel values ​​more accurately than the row adjustment coefficient.

[0037] (3) The present invention can project equal optical paths to multiple color multi-line sensors through the spectral technology, and delay the installation of each color multi-line sensor in sequence, so as to achieve a splicing effect of different sensors in the same space and realize the fusion data output of different pixel positions in each RGB component.

[0038] (4) Based on the characteristics of color multi-line sensors, the present invention can continuously image the target object multiple times on different pixel rows and combine with a flexible TDI overlay method to realize the HDR function. Compared with the HDR method of collecting multiple images for synthesis, it has the following advantages: the output frame rate / line frequency is not affected, while the frame rate / line frequency of the traditional multi-image synthesis method will be reduced according to the proportion of the number of synthesized images, such as the frame rate is halved in the case of a 2-image synthesis method; the temporal and spatial correlation is very high, and the temporal and spatial consistency of the synthesized image is good, while in the traditional multi-image synthesis algorithm, the acquisition time of multiple images is quite different, and the temporal and spatial consistency is poor. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0040] Figure 1 is a flow chart of the processing method in the present invention;

[0041] Figure 2 is a schematic diagram of a four-wire sensor in the present invention;

[0042] Figure 3 is a schematic diagram of a twelve-line color sensor in the present invention;

[0043] Figure 4 It is a flow chart of an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the combined acquisition of three color sensors in the present invention;

[0045] Figure 6 This is a schematic diagram of the combined acquisition of three color sensors in the present invention;

[0046] Figure 7 It is a principle diagram of pixel correction in the Y direction in the present invention;

[0047] Figure 8 It is a schematic diagram of the pixel correction principle in the X direction of the present invention;

[0048] Fig. 9 It is a schematic diagram of TDI data cache in the present invention;

[0049] Fig.10 It is a structural block diagram of the implementation system in the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example 1

[0051] A multi-line sensor contains multiple pixel rows, such as Figure 2 The figure shows the internal structure of the currently more common 4-line sensor. There are four rows of photosensitive pixels in the multi-line sensor, and the row spacing is the same as the length and width of the pixel. Compared with ordinary single-line sensors, multiple rows of pixels in the multi-line sensor can be exposed at the same time, solving the problem of insufficient responsiveness under high line frequency (low exposure time). The usual usage of TDI cameras designed based on multi-line sensors is to perform TDI (time delay integration) superposition of multiple exposure pixel rows according to the imaging time, such as Figure 2 As shown, 4 lines are superimposed into 1 line output, that is, the responsiveness of a 4-line TDI camera is 4 times that of a single-line camera. This is the fundamental feature of a TDI camera.

[0052] The main purpose of HDR, or high dynamic range, is to avoid the problem of premature overexposure of some pixels in the image, resulting in information loss, while other pixels have insufficient response values. There are many ways to achieve this, including a more complex real-time adjustment of the electron well depth (which can be achieved by adjusting the reset voltage during exposure), a simple fake implementation using different ADC gains, or a method of acquiring and synthesizing multiple images.

[0053] like Figure 1 As shown, this embodiment first provides a high dynamic range image processing method based on a multi-line sensor, and the specific method is as follows:

[0054] The pixel rows of each RGB component in the color multi-line sensor are divided so that each RGB component forms at least two groups of photosensitive units, each group of photosensitive units includes several consecutive pixel rows, and the exposure time of each pixel row is independently adjusted.

[0055] As for the number of multi-line sensors, it is possible to group multiple pixel rows of a single multi-line sensor, or to splice pixel rows of multiple multi-line sensors to form a special case equivalent to one multi-line sensor. Continuous pixel rows represent multiple pixel rows corresponding to the same position of the object to be measured in the direction of motion based on continuous time.

[0056] The scanning data of each pixel row is obtained, and the scanning data in all photosensitive units in each RGB component are superimposed to form TDI data corresponding to each group of photosensitive units.

[0057] As a specific implementation of scanning data superposition, the scanning data of each pixel row in each group of photosensitive units can be processed by summing or averaging. Summing means adding and averaging multiple pixel values ​​at the same pixel position in the same group, which can be used to increase responsiveness, while averaging means adding and averaging multiple pixel values ​​at the same pixel position in the same group, which can be used to increase the signal-to-noise ratio and reduce the impact of noise to improve image quality.

[0058] As embodiment 1, for a color multi-line sensor with a TDI coefficient greater than or equal to two, Figure 3 Taking a 12-line color sensor with a TDI coefficient of 4 as an example, the TDI coefficient indicates the number of pixel rows contained in each RGB component in the same color multi-line sensor. The TDI superposition mode is changed from 4 lines in one to 1+3 superposition mode. Taking the R component as an example, R1 is the first group, R2, R3, and R4 are the second group, and R2, R3, and R4 in the second group are superimposed as line X according to TDI. When the sensor single-line exposure / gain settings are the same, the response ratio of R1 and line X is 1:3. At the same time, there is a high temporal and spatial correlation between R1 and line X, that is, the photographed target has exactly the same morphological characteristics, which provides good conditions for the next HDR synthesis.

[0059] In addition to the 1+3 superposition method, there is also a 2+2 superposition method, that is, R1 and R2 are the first group, R3 and R4 are the second group, and TDI superposition is completed in each group and line Y (first group) and line Z (second group) are output. At this time, when the sensor single-line exposure / gain settings are the same, the response ratio of line Y and line Z is 1:1.

[0060] For color multi-line sensors with a TDI coefficient of 8 or 16, there are more superposition methods, which are not listed here one by one.

[0061] It can be seen that the response ratio coefficient obtained by simply changing the TDI superposition method is fixed, that is, once the grouping method is determined, the response ratio coefficient will no longer change. Although it can cope with some relatively fixed scenes, it cannot adapt to more flexible scenes. At this time, the function of independently adjusting the exposure time of each pixel row provided by the multi-line sensor itself can be used to achieve arbitrary response difference. Figure 3 Take the 12-line color sensor in as an example, let the superposition mode be 1+3, the exposure time of R1 be T1, and the exposure time of R2, R3, and R4 be the same, which is T2. According to the above, the response ratio of R1 and line X is K1:K2=T1:3T2. By appropriately adjusting the exposure time lengths of T1 and T2, any response proportional coefficient can be achieved.

[0062] The pixel values ​​of all pixel positions in each group of TDI data are collected, and the corresponding pixel value adjustment coefficients are calculated.

[0063] As a specific implementation method for calculating the pixel value adjustment coefficient, the closer the pixel values ​​corresponding to all pixel positions in each group of TDI data are to half of the maximum pixel value, the larger the pixel value adjustment coefficient is, and vice versa.

[0064] In order to meet the following calculation requirements, image normalization is performed before collecting the pixel values ​​of all pixel positions in each group of TDI data. The following is an explanation using a specific case:

[0065] From the above, it can be seen that after dividing the pixel rows of the multi-line sensor, at least two groups of photosensitive units are formed. Taking two groups of photosensitive units as an example, according to the exposure time of each group of photosensitive units, the group with a smaller exposure time is defined as a low-exposure image, and the group with a larger exposure time is defined as a high-exposure image.

[0066] Pixel value adjustment factor for low exposure images Determined by the pixel value, Represents the pixel value adjustment coefficient of the TDI data of the corresponding group of low-exposure images at the (i, j) coordinate, Represents the pixel value of the TDI data of the corresponding group of low-exposure images at the (i, j) coordinate, When it is less than 0.5, the smaller the pixel value, the smaller the pixel value adjustment coefficient. The pixel value adjustment coefficients greater than 0.5 are uniformly set to 1, which can be obtained by the following piecewise function:

[0067]

[0068] Pixel value adjustment factor for high exposure images It is also determined by the pixel value. Represents the pixel value adjustment coefficient of the TDI data of the corresponding group of high exposure images at the (i, j) coordinate, Represents the pixel value of the TDI data of the corresponding group of high exposure images at the (i, j) coordinate, When it is less than 0.5, the pixel value adjustment coefficient is uniformly set to 1. When it is greater than 0.5, the larger the pixel value, the smaller the pixel value adjustment coefficient, which is obtained by the following piecewise function:

[0069]

[0070] In addition to two groups of photosensitive units, three or more groups of photosensitive units may be formed. For this purpose, the pixel value adjustment coefficient of the Mth group of TDI data at the (i, j) coordinate can be constructed by establishing a lookup table. The pixel value at the (i, j) coordinate of the Mth group of TDI data The mapping relationship satisfies that the closer the pixel value is to half of the maximum pixel value, the larger the pixel value adjustment coefficient is, and vice versa.

[0071] The exposure time corresponding to each group of TDI data is obtained, and the exposure reference time is set. The corresponding pixel adjustment value is calculated according to the ratio of the exposure reference time to the exposure time of all pixel positions in each group of TDI data, so as to adjust the pixel values ​​of all pixel positions in each group of TDI data to the same energy level.

[0072] Since the data collected by each group of TDI data at different exposure times are at different energy levels, they need to be adjusted to the same energy level according to the exposure time parameter. To prevent value overflow (greater than 1), a smaller exposure reference time t can be set. K , based on the exposure reference time and the exposure time t of all pixel positions in each set of TDI data M The corresponding pixel adjustment value is obtained by ratio calculation, which is specifically implemented by the following formula:

[0073] Among them, it is assumed that there are N groups of photosensitive units. is the pixel value of the Mth group of TDI data at the (i, j) coordinate, for The corresponding pixel adjustment value.

[0074] For example, taking the two groups of photosensitive units mentioned above as an example, the two groups of photosensitive units correspond to the low-exposure image and the high-exposure image respectively, and the exposure time of the group corresponding to the low-exposure image is t L , the exposure time of the high exposure image group is t H , the exposure reference time t K Set to t L , that is, the high-exposure image can be directly reduced to the energy level of the low-exposure image. Represents the pixel value of the TDI data of the corresponding group of high exposure images at the (i, j) coordinate, for The corresponding pixel adjustment value, for The corresponding pixel adjustment value remains unchanged, and the specific formula is as follows:

[0075]

[0076]

[0077] Obtain the pixel value adjustment coefficient and pixel adjustment value of the same pixel position of all TDI data in each RGB component, and obtain the first fused pixel value of the same pixel position by weighted calculation , thereby obtaining row image data with high dynamic range. The fusion process is as follows:

[0078]

[0079] in, is the pixel value adjustment coefficient of the Mth group of TDI data at the (i, j) coordinate, is the pixel adjustment value of the Mth group of TDI data at the (i, j) coordinate, where e is a minimum value to prevent the denominator from being zero.

[0080] The following also takes the two groups of photosensitive units mentioned above as an example. The two groups of photosensitive units correspond to the low-exposure image and the high-exposure image respectively. Represents the pixel value adjustment coefficient of the TDI data of the corresponding group of high exposure images at the (i, j) coordinate, Represents the pixel adjustment value of the TDI data of the corresponding group of high exposure images at the (i, j) coordinate, Represents the pixel value adjustment coefficient of the TDI data of the corresponding group of low-exposure images at the (i, j) coordinate, Represents the pixel adjustment value of the TDI data of the corresponding group of low-exposure images at the (i, j) coordinate, then:

[0081]

[0082] Collect the first fused pixel value of each pixel position , the second fused pixel value is obtained by tone mapping , thereby acquiring and outputting row image data with a low dynamic range.

[0083] Since most existing display devices can only display LDR images, these devices usually have a limited brightness range and color depth and cannot directly display the wide brightness range and rich color details in HDR images. Therefore, the purpose of tone mapping is to map the brightness, color temperature, dynamic range and other characteristics of HDR images to the standard range of LDR images, while retaining the visual effects and detail information of bright and dark areas of the original image as much as possible, so as to facilitate display on ordinary monitors.

[0084] As a specific implementation of tone mapping, tone mapping can be processed by using any one of gamma transformation, log domain transformation and tone mapping function.

[0085] The gamma transform obtains the tone-mapped image by the following formula: :

[0086] In the formula .

[0087] The log domain transformation first maps the image to the log domain with base 2, and then normalizes it:

[0088]

[0089] Where f N () represents the normalization function.

[0090] The tone mapping function adds adaptive histogram equalization on the basis of log domain transformation. Adaptive histogram equalization calculates the local histogram of the image and then redistributes the brightness to improve the local contrast of the image. In addition, the function also adds brightness and saturation adjustment. Brightness adjustment maps the value range of the image result to a given brightness range. By default, the result brightness is not adjusted.

[0091] The specific implementation process is as follows Figure 4 As shown in the figure, after the camera is powered on, it starts to collect images, configures HDR parameters and performs indirect parameter calculations, then configures the sensor and adjusts the exposure timing to obtain multi-line image data, configures the FPGA to implement the TDI overlay algorithm and HDR synthesis algorithm, and finally realizes the data output of the HDR image.

[0092] The implementation idea of ​​the above method is based on the characteristics of multi-line sensors. It can continuously image the target object multiple times on different pixel rows and combine with a flexible TDI overlay method to realize the HDR function. Compared with the HDR method of acquiring multiple images for synthesis, it has the following advantages: the output frame rate / line frequency is not affected, while the frame rate / line frequency of the traditional multi-image synthesis method will be reduced according to the proportion of the number of synthesized images, such as the frame rate is halved in the case of 2-image synthesis; the time-space correlation is very high, and the time-space consistency of the synthesized image is good, while in the traditional multi-image synthesis algorithm, the acquisition time of multiple images is quite different, and the time-space consistency is poor. Example 2

[0093] The color multi-line sensor in Example 1 may be installed in a single or multiple combination. At least two color multi-line sensors may be installed, each with a TDI coefficient of one, and the incident light may be divided into multiple paths and projected to each color multi-line sensor with equal optical path, and each color multi-line sensor may be delayed by one pixel position in turn along the moving direction of the object to be measured; wherein the pixel interval of the color multi-line sensor is a pixel width, and the number of the color multi-line sensors is N, where N=a+1;

[0094] The pixel rows of the same component in each color multi-line sensor are summarized in sequence and divided into at least two groups of photosensitive units to calculate the first fused pixel values ​​of different pixel positions in each RGB component, thereby obtaining and outputting RGB row image data with a high dynamic range.

[0095] For a color multi-line sensor with a TDI coefficient equal to one, such as a three-line color sensor, in traditional applications, since its components cannot be grouped and superimposed, it is naturally impossible to use it to realize the HDR function. The present invention effectively utilizes the conventional color multi-line sensor with a TDI coefficient equal to one, combines multiple color multi-line sensors with spectroscopic technology, realizes the data output of HDR images, can effectively utilize existing camera resources, and avoids waste of camera resources.

[0096] like Figure 5 As shown, the color multi-line sensor includes a set of continuous RGB components, namely, a three-line color sensor. The pixel interval of the color multi-line sensor is 2 pixel widths, and the number of color multi-line sensors is 3. The incident light beam is divided into three by a three-point prism and projected onto three color multi-line sensors. The design requires that the optical path of the incident light beam reaching the three color multi-line sensors is the same to ensure the same imaging magnification, and the three color multi-line sensors are delayed one pixel position in turn along the moving direction of the object to be measured, which means that the three color multi-line sensors are staggered in space.

[0097] The moving speed of the object to be tested is 1 / T, and the acquisition interval of each color multi-line sensor is T. Assuming that the positions of the objects to be tested are numbered 1, 2, 3, 4, 5, 6, etc., and each color multi-line sensor collects synchronously, the imaging of the object to be tested on the three color multi-line sensors is as follows: Figure 5 As shown, three consecutive pixel rows of each RGB component are grouped and superimposed, for example, in a 1+2 superposition manner, so as to calculate the first fused pixel value and the second fused pixel value of each RGB component, and output the fused RGB data. Example 3

[0098] The difference from the second embodiment is that the color multi-line sensor includes at least two groups of continuous RGB components. For example, the six-line color sensor is distributed in RGBRGB. Then, the imaging of the object to be tested on the three color multi-line sensors is as follows: Figure 6 As shown, taking the R component as an example, every three consecutive R / G / B data are superimposed to form a group of TDI data, forming two groups in total. If three consecutive RGB components are included, three groups are formed. Finally, each group of R / G / B data is fused to obtain the first fused pixel value and the second fused pixel value corresponding to the R / G / B component, and finally the fused RGB data is output. Example 4

[0099] For the installation scenario of multiple color multi-line sensors in Example 2, since they need to be installed staggered, installation deviation may occur. For the pixel deviation caused by the installation deviation, a weighted average algorithm can be used to correct the pixels. The specific process is as follows:

[0100] Image information collected by each color multi-line sensor is extracted and compared to obtain a pixel deviation value, wherein the exposure time of each color multi-line sensor is the same.

[0101] like Figure 5 As shown, since the exposure time of each color multi-line sensor is the same, although one pixel position is delayed in sequence to form a staggered installation, they should be basically consistent after alignment. Therefore, if there is a difference, it is equivalent to being considered as a difference caused by pixel offset. The acquisition of such image difference can collect a standard template image to calculate the difference in the X and / or Y direction. For example, the image information collected by one of the color multi-line sensors can be used as a reference image to calculate the difference between the image information collected by other sensors and the reference image. For this, the pixel deviation value δx and / or δy can be obtained by edge position fitting.

[0102] It should be noted that the above correction is local, that is, it is only effective when the pixel deviation value δx and / or δy is small. Once the difference value is large, it is actually equivalent to an interpolation operation, and the meaning and effectiveness of the pixel correction are lost.

[0103] Therefore, in order to ensure the effectiveness of the sub-pixel alignment compensation method, it also includes: comparing the difference threshold to determine the size of the physical position difference of each sensor; if it is less than or equal to the difference threshold, a weighted average algorithm is used to perform pixel correction on the color multi-line sensor to be corrected; otherwise, each sensor is driven at the same time to collect image information; the difference threshold is selected as 0.5 pixels, that is, the value range of δx and / or δy should be less than or equal to 0.5 pixels, so as to ensure the effectiveness of the weighted average algorithm when performing pixel correction on the color multi-line sensor to be corrected.

[0104] The pixel deviation value of each color multi-line sensor is obtained, and the color multi-line sensor to be corrected is identified.

[0105] The weighted average algorithm is used to perform pixel correction on the corrected color multi-line sensor.

[0106] Specific as Figure 7 and Figure 8 As shown, taking the R component as an example, the pixel correction principle diagram corresponding to δy and δx is shown respectively, R1 y , R2 y 、R3 y Respectively represent the actual pixel values ​​collected corresponding to the positions 1, 2, and 3 of the object to be tested along the Y direction, R1 x , R2 x 、R3 x They represent the actual pixel values ​​collected at positions 1, 2, and 3 of the object to be measured along the X direction, respectively. is the pixel correction value along the Y direction at position 2, It is the pixel correction value of position 2 along the X direction, including the following three cases:

[0107] Case 1: There is only pixel offset in the Y direction, then the corresponding pixel correction formula is:

[0108]

[0109] Among them, β=δy, and 0≤β≤0.5.

[0110] Case 2: There is only pixel offset in the X direction, then the corresponding pixel correction formula is:

[0111]

[0112] Among them, α=δx, and 0≤α≤0.5.

[0113] Case 3: There is pixel offset in both the X and Y directions. In this case, the following four correction methods can be used:

[0114] First correction method: First, according to the scheme of case 1, the pixel offset in the Y direction is corrected to obtain the pixel correction value of each pixel point of the color multi-line sensor to be corrected along the Y direction, and this is used as R2 in case 2. x 、R3 x , the final correction data is calculated using the correction formula of case 2 and output, thereby completing the pixel correction in the X direction and the Y direction.

[0115] Second correction method: First, according to the second solution, the pixel offset in the X direction is corrected to obtain the pixel correction value of each pixel point of the color multi-line sensor to be corrected along the X direction, and this is used as R2 in the first solution. y 、R3 y , the final correction data is calculated using the correction formula of case 1 and output, thereby completing the pixel correction in the X direction and the Y direction.

[0116] The third correction method: Correct the pixel offset in the Y direction according to the first solution to obtain According to the second solution, the pixel offset in the X direction is corrected to obtain , will get and The average is calculated and output as the final correction data, thereby completing the pixel correction in the X direction and the Y direction.

[0117] Fourth correction method: Correct the pixel offset in the Y direction according to the first solution to obtain According to the second solution, the pixel offset in the X direction is corrected to obtain , and then calculate according to the following formula:

[0118]

[0119] is the combined pixel correction value of position 2 along the Y direction and the X direction, and the obtained It is output as the final correction data, thereby completing the pixel correction in the X direction and the Y direction. Example 5

[0120] As can be seen from Example 1, as a specific implementation method of scanning data superposition, the scanning data of each pixel row in each group of photosensitive units can be summed or averaged for processing, so that the TDI data corresponding to each group of photosensitive units can be obtained. As can be seen from the background technology, the existing low-order TDI camera itself usually does not have the TDI function, but completes the TDI operation in the FPGA after all the pixel rows are read out, that is, the image data generated for each row of pixels are stored in the storage space at different addresses in the FPGA. Then, when the pixel row scanning data for a certain position of the object to be tested (usually the detection area corresponding to a row of pixel rows) is all stored in the FPGA, the scanning data of the corresponding pixel rows are superimposed by TDI, that is, the above-mentioned scanning data are superimposed to form a superimposed row image.

[0121] However, the above scanning data superposition generation method requires a large amount of FPGA storage space. In some work scenarios, it is necessary to give up using some FPGAs with low storage resources and select FPGAs with larger cache resources as back-end processors. Therefore, the current TDI cache method not only increases the cost and power consumption in the machine vision inspection process, but also makes FPGAs with low cache resources unusable, resulting in a waste of resources.

[0122] Taking the R component in the color multi-line sensor as an example, assuming that one group of photosensitive units contains 4 rows of pixels, namely R1-R4, and the row interval is the same as the pixel size. In the normal single-light mode, the ratio of the line frequency to the object movement rate is 1:1, and the object movement direction is from top to bottom. Taking the pixel row R4 as the reference, the image data output by R1 needs to cache 7 rows, the image data output by R2 needs to cache 5 rows, and the image data output by R3 needs to cache 3 rows. When the 7th row of image data of R4 is input to the FPGA, the 1st row of image data of R1, the 3rd row of image data of R2, the 5th row of image data of R3 and the 7th row of image data of R4 are superimposed to complete the TDI operation. The number of rows corresponding to the total cache space is 1+3+5+7=16 rows. With the increase of the TDI superposition order, the demand for FPGA cache resources under the existing multi-line cache method is very high, which adds great pressure on cost and power consumption.

[0123] Therefore, when the number of pixel rows in the photosensitive unit is greater than or equal to two, in order to solve the cache resource problem caused by the superposition of scan data, the present invention also provides a TDI data cache processing method, which specifically includes the following steps:

[0124] Step 1: Obtain the first data and the second data output by each group of photosensitive units, and store the first data line by line in cache spaces at different addresses.

[0125] The first data represents image data generated by the first pixel row in the photosensitive unit collecting different positions of the object to be tested; the second data represents image data generated by non-first pixel row in the photosensitive unit collecting different positions of the object to be tested.

[0126] Step 2: Read the stored data to be superimposed, superimpose it with the second data at the same position, and form the data to be superimposed for the next superimposition process, and re-store it in the original cache space of the data to be superimposed; repeat the superposition process until the first data at the same position is superimposed with all the second data, and output the image data after the superposition is completed.

[0127] The data to be superimposed include first data and image data obtained by superimposing the first data and the second data.

[0128] If the average value of the scanned data of each pixel row in the photosensitive unit is to be calculated, the average value of the superimposed image data can be calculated.

[0129] Based on the above processing, the present application realizes the reuse of the same cache space in the TDI data caching process by superimposing the data to be superimposed containing the same position in sequence and caching them into the cache space of the corresponding first data. Compared with the existing method of caching and superimposing multiple rows of pixel data separately, the present application can effectively reduce the cache resource requirements and can greatly reduce the pressure on the FPGA cache space requirements during the TDI data generation process.

[0130] For step 1, the first row of pixels represents the pixel row that first obtains the image information of the detected position of the object to be tested. Fig. 9 For example, R1 is the first row of pixels, and R2, R3, and R4 are all non-first row of pixels. Similarly, the image data of R1 belongs to the first data, and the image data of R2 after the third exposure, the image data of R3 after the fifth exposure, and the image data of R4 after the seventh exposure all belong to the second data. Row-by-row storage means that under the same exposure light source, the image data generated by the first row of pixels is used as a fixed unit, and the image data generated at different positions are stored separately.

[0131] For step 2, the superposition process stops until all rows of pixels corresponding to the same position of the object to be tested have completed TDI superposition. Fig. 9For example, the image data of row R1 after the first exposure, the image data of row R2 after the third exposure, the image data of row R3 after the fifth exposure, and the image data of row R4 after the seventh exposure are all superimposed and cached, and the superimposed image data are output to the next processing module for the next step of data processing or data transmission.

[0132] The data to be superimposed include three types: first data, image data obtained by superimposing the first data and a single second data, and image data obtained by superimposing the first data and multiple second data. Specifically, when the first data of the first row of pixels that have been stored are superimposed with the second data of the second row of pixels, the data to be superimposed corresponds to the first data; when the first data and the second data of the second row of pixels are superimposed and stored, and then superimposed with the second data corresponding to the third row of pixels again, the data to be superimposed corresponds to the image data obtained by superimposing the first data and the single second data; when the first data and multiple second data of multiple rows of pixels are superimposed and stored, and then superimposed with another second data corresponding to the next row of pixels again, the data to be superimposed corresponds to the image data obtained by superimposing the first data and multiple second data.

[0133] In addition, for the same position of the object to be tested, the TDI superposition process is achieved by accumulating the image data of the first pixel row (i.e., the first data) and the image data of the remaining pixel rows (the second data). In the first superposition process, the data to be superimposed are all the first data, so the original storage space of the data to be superimposed is represented as the storage space corresponding to the original address of the first data at the same position.

[0134] In some embodiments, when the color multi-line sensor simultaneously outputs image data of two or more pixel rows in one line cycle, such as image data of R1 and R2 are output simultaneously, at this time, the RAM must complete the writing of the R1 image data on the one hand, and process the reading and writing of the R2 image data on the other hand.

[0135] In view of this situation, in the TDI cache solution provided by this application, the storage space of the FPGA is set up in a single-row cascade to support simultaneous operations on 2 or more rows of pixel row data at the same time. Specifically, a single row means that the total storage space of the FPGA is composed of multiple independent RAMs, each of which can cache the image data of a single row of pixels. When the application needs to store a certain number of pixel rows, at least that many RAMs are set in the FPGA, that is, one RAM stores one row. Cascading means the cascading of data between multiple RAMs. When a new row of pixel image data comes from the front end, the previous row of data read out by the previous RAM is simultaneously written into the next RAM, that is, the image data of multiple rows of pixels are translated in multiple RAMs.

[0136] It is worth noting that when the storage space of the FPGA adopts a single-row cascade setting, it only changes the connection relationship of the memory cells in the RAM and the processing process of writing the image data into the RAM, while the processing process of the TDI cache method is consistent with the above and no adjustment is required. Example 6

[0137] like Fig.10 As shown, in the second aspect of the present invention, the present invention also provides a high dynamic range image processing system based on a multi-line sensor, comprising:

[0138] The parameter adjustment module is used to divide the pixel rows of each RGB component in the color multi-line sensor so that each RGB component forms at least two groups of photosensitive units, each group of photosensitive units includes several consecutive pixel rows, and the exposure time of each pixel row is adjusted independently.

[0139] The data superposition module is used to obtain the scanning data of each pixel row, complete the scanning data superposition in all photosensitive units in each RGB component, and form TDI data corresponding to each group of photosensitive units.

[0140] The coefficient calculation module is used to collect the pixel values ​​of all pixel positions in each group of TDI data and calculate the corresponding pixel value adjustment coefficient.

[0141] The energy unification module is used to obtain the exposure time corresponding to each group of TDI data, set the exposure reference time, and calculate the corresponding pixel adjustment value based on the ratio of the exposure reference time to the exposure time of all pixel positions in each group of TDI data, so as to adjust the pixel values ​​of all pixel positions in each group of TDI data to the same energy level.

[0142] The weighted calculation module is used to obtain the pixel value adjustment coefficient and pixel adjustment value of the same pixel position of all TDI data in each RGB component, and obtain the first fused pixel value of the same pixel position by weighted calculation, so as to obtain row image data with high dynamic range.

[0143] The tone mapping module collects the first fused pixel value of each pixel position, obtains the second fused pixel value through tone mapping, thereby obtaining and outputting row image data with a low dynamic range.

[0144] The specific operation method of the processing system can be implemented by referring to the above processing method. Example 7

[0145] The third aspect of the present invention further provides a computer-readable storage medium, comprising a computer program, wherein the computer program implements the above-mentioned processing method when executed by a processor.

[0146] In practical applications, the computer-readable storage medium may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device.

[0147] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0148] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0149] Computer program code for performing the operation of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0150] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0151] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A high dynamic range image processing method based on a multi-line sensor, characterized in that: include: Dividing the pixel rows of each RGB component in the color multi-line sensor so that each RGB component forms at least two groups of photosensitive units, each group of photosensitive units includes a plurality of consecutive pixel rows, and the exposure time of each pixel row is independently adjusted; Obtain the scan data of each pixel row, complete the superposition of the scan data in all photosensitive units in each RGB component, and form the TDI data corresponding to each group of photosensitive units; Collect pixel values ​​of all pixel positions in each group of TDI data and calculate corresponding pixel value adjustment coefficients; Obtaining the exposure time corresponding to each group of TDI data, setting the exposure reference time, and calculating the corresponding pixel adjustment value according to the ratio of the exposure reference time to the exposure time of all pixel positions in each group of TDI data, so as to adjust the pixel values ​​of all pixel positions in each group of TDI data to the same energy level; The corresponding pixel adjustment value is calculated, which is specifically implemented by the following formula: Among them, it is assumed that there are N groups of photosensitive units. is the pixel value of the Mth group of TDI data at the (i, j) coordinate, for The corresponding pixel adjustment value, t K is the exposure reference time, t M is the exposure time of all pixel positions in each set of TDI data; Obtaining the pixel value adjustment coefficient and the pixel adjustment value of the same pixel position of all TDI data in each RGB component, and performing weighted calculation to obtain the first fused pixel value of the same pixel position, thereby obtaining row image data with a high dynamic range; The weighted calculation is as follows: in, is the first fused pixel value at the same pixel position, is the pixel value adjustment coefficient of the Mth group of TDI data at the (i, j) coordinate, is the pixel adjustment value of the Mth group of TDI data at the (i, j) coordinate, and e is a minimum value.

2. The high dynamic range image processing method based on a multi-line sensor according to claim 1, characterized in that: The scan data superposition is processed by summing or averaging the scan data of each pixel row in each group of photosensitive units.

3. The high dynamic range image processing method based on a multi-line sensor according to claim 1, characterized in that: The closer the pixel values ​​corresponding to all pixel positions in each group of TDI data are to half of the maximum pixel value, the larger the pixel value adjustment coefficient is, and vice versa.

4. The high dynamic range image processing method based on a multi-line sensor according to any one of claims 1 to 3, characterized in that: Image normalization is performed before collecting the pixel values ​​of all pixel positions in each set of TDI data.

5. The high dynamic range image processing method based on a multi-line sensor according to claim 4, characterized in that: The TDI coefficient of the color multi-line sensor is greater than or equal to two, and the number of the color multi-line sensors is one.

6. The high dynamic range image processing method based on a multi-line sensor according to claim 5, characterized in that: When the number of pixel rows in the photosensitive unit is greater than or equal to two, the method further includes: Acquire the first data and the second data output by each group of photosensitive units, and store the first data row by row in the cache space with different addresses; wherein the first data represents the image data generated by the first row of pixels in the photosensitive unit collecting different positions of the object to be tested; and the second data represents the image data generated by the non-first row of pixels in the photosensitive unit collecting different positions of the object to be tested; The stored data to be superimposed is read, superimposed with the second data at the same position, and the data to be superimposed for the next superimposition process is formed, and is stored again in the original cache space of the data to be superimposed; the superimposition process is repeated until the first data at the same position is superimposed with all the second data, and the image data after the superposition is completed is output; wherein the data to be superimposed includes the first data, and the image data after the first data and the second data are superimposed.

7. The high dynamic range image processing method based on a multi-line sensor according to claim 4, characterized in that: At least two color multi-line sensors are installed, each color multi-line sensor has a TDI coefficient of one, and the incident light is divided into multiple paths and projected to each color multi-line sensor with equal optical path, and each color multi-line sensor is delayed by one pixel position in turn along the moving direction of the object to be measured; wherein the pixel interval of the color multi-line sensor is a pixel width, and the number of the color multi-line sensors is N, N=a+1; The pixel rows of the same component in each color multi-line sensor are summarized in sequence and divided into at least two groups of photosensitive units to calculate the first fused pixel values ​​of different pixel positions in each RGB component, thereby obtaining and outputting RGB row image data with a high dynamic range.

8. The high dynamic range image processing method based on a multi-line sensor according to claim 7, characterized in that: Also includes: Extracting image information collected by each color multi-line sensor and comparing them to obtain a pixel deviation value, wherein the exposure time of each color multi-line sensor is the same; Obtain pixel deviation values ​​of each color multi-line sensor and identify the color multi-line sensor to be corrected; The weighted average algorithm is used to perform pixel correction on the corrected color multi-line sensor.

9. A high dynamic range image processing system based on a multi-line sensor, characterized in that: include: A parameter adjustment module, used for dividing the pixel rows of each RGB component in the color multi-line sensor so that each RGB component forms at least two groups of photosensitive units, each group of photosensitive units includes a number of consecutive pixel rows, and the exposure time of each pixel row is adjusted independently; The data superposition module is used to obtain the scan data of each pixel row, complete the scan data superposition of all photosensitive units in each RGB component, and form the TDI data corresponding to each group of photosensitive units; A coefficient calculation module is used to collect pixel values ​​of all pixel positions in each group of TDI data and calculate the corresponding pixel value adjustment coefficient; An energy unification module is used to obtain the exposure time corresponding to each group of TDI data, set the exposure reference time, and calculate the corresponding pixel adjustment value according to the ratio of the exposure reference time to the exposure time of all pixel positions in each group of TDI data, so as to adjust the pixel values ​​of all pixel positions in each group of TDI data to the same energy level; The weighted calculation module is used to obtain the pixel value adjustment coefficient and pixel adjustment value of the same pixel position of all TDI data in each RGB component, and obtain the first fused pixel value of the same pixel position by weighted calculation, so as to obtain row image data with high dynamic range.

10. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program is executed by a processor, the processing method according to any one of claims 1 to 8 is implemented.

Images