High frame rate camera and imaging method

Abstract

The embodiment of the present application provides a high frame rate camera and imaging method. For each image sensor, the image sensor is used to collect raw data based on a preset trigger signal, and transmit the raw data to the processing unit corresponding to the image sensor. For each processing unit, the processing unit is used to correct the received raw data based on the corresponding ISP parameters to obtain an initial image sequence, and transmit it to a post-processing unit. The post-processing unit is used to rearrange all the initial image sequences in time order to obtain a target image sequence. Through precise time synchronization and data acquisition, accurate image correction processing, and orderly image sequence reorganization, efficient and high-quality processing of the raw data collected by multiple image sensors is achieved, thereby generating a high-quality target image sequence.

Description

Technical Field

[0001] The present application relates to the field of image acquisition technology, and in particular to a high frame rate camera and an imaging method. Background Art

[0002] A high frame rate camera is a camera that can capture high-speed dynamic scenes and can collect image data at a very high frequency, usually at a speed of tens to hundreds of frames per second, which is much higher than the frame rate of ordinary cameras. Due to the use of advanced sensors and image processing technology, high frame rate cameras can capture clearer images and more details. Due to the high frame rate, the amount of image data generated by high frame rate cameras in the same time is also larger, which places higher requirements on storage and transmission capabilities.

[0003] Currently, image sensors for high frame rate cameras need to capture and transmit image data quickly, and high resolution means more data needs to be processed. When the resolution increases, the data output rate and processing power of the sensor become limiting factors. The simultaneous realization of high resolution and high frame rate requires the sensor to have higher sensitivity and faster response speed. However, these performance improvements are limited by material science, manufacturing process and other aspects. And once the hardware solution of a high frame rate camera is determined, its performance parameters such as resolution and frame rate are difficult to adjust significantly. This limits the flexibility and scalability of the camera.

[0004] In summary, how to ensure the high frame rate of high frame rate cameras while improving scalability is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a high frame rate camera and an imaging method to solve the problem of how to ensure the high frame rate of the high frame rate camera while improving the scalability.

[0006] In a first aspect, an embodiment of the present application provides a high frame rate camera, comprising: an acquisition unit group and a processing unit group, wherein the acquisition unit group comprises N image sensors, and the processing unit group comprises M processing units and a post-processing unit, wherein N is an integer greater than or equal to 2, and M is an integer greater than or equal to 1;

[0007] For each image sensor, the image sensor is used to collect raw data based on a preset trigger signal, and transmit the raw data to a processing unit corresponding to the image sensor; wherein, for the first N-1 image sensors, the time difference between the trigger signals of adjacent image sensors is a first preset time, and the time difference between the trigger signals of the Nth image sensor and the N-1th image sensor is a second preset time, the first preset time is calculated based on the frame rate of each image sensor, and the second preset time is calculated based on the first preset time and the frame rate;

[0008] For each processing unit, the processing unit is used to perform correction processing on the raw data based on the corresponding ISP parameters to obtain an initial image sequence and transmit it to the post-processing unit, wherein the ISP parameters are determined based on the calibration parameters of the image sensor corresponding to the raw data and the time difference between the trigger signal corresponding to the image sensor and the trigger signal corresponding to the previous image sensor;

[0009] The post-processing unit is used to rearrange all initial image sequences in time order to obtain a target image sequence.

[0010] In a possible implementation manner, the ISP parameters include a white balance adjustment value and an exposure adjustment value, and the processing unit performs correction processing on the raw data based on the corresponding ISP parameters to obtain an initial image sequence, including:

[0011] For each processing unit, the processing unit adjusts the raw data based on the white balance adjustment value to obtain adjusted raw data, wherein the white balance adjustment value is determined based on a calibration parameter of an image sensor corresponding to the raw data and a time difference between a trigger signal corresponding to the image sensor and a trigger signal corresponding to a previous image sensor;

[0012] The processing unit adjusts the adjusted raw data based on the exposure adjustment value to obtain the initial image sequence, wherein the exposure adjustment value is determined based on a time difference between a trigger signal corresponding to the image sensor and a trigger signal corresponding to a previous image sensor.

[0013] In a possible implementation, it further includes:

[0014] For each time period of the original data, determining a white balance statistic based on the average values ​​of the three color channels of the original data corresponding to the time period;

[0015] Determine a white balance calculation value corresponding to the time period according to the white balance statistics and a white balance calculation value of a previous image sensed by the image sensor corresponding to the raw data;

[0016] A white balance adjustment value corresponding to the time period is determined based on the white balance calculation value corresponding to the time period and a calibration parameter of the image sensor corresponding to the raw data.

[0017] In a possible implementation, it further includes:

[0018] For each time period of the original data, determining an exposure statistic value based on brightness information of the original data corresponding to the time period;

[0019] Determine the exposure calculation value corresponding to the time period according to the exposure statistics and the exposure calculation value of the previous image sensing of the image sensor corresponding to the raw data;

[0020] The calculated exposure value corresponding to the time period is determined as the exposure adjustment value corresponding to the time period.

[0021] In a possible implementation, it further includes:

[0022] For each image sensor, a background gray image pre-taken under a fixed light source environment is divided into a plurality of blocks;

[0023] Select the middle P blocks as the target position, where P is an integer;

[0024] Extracting the three-color channel information corresponding to the target position;

[0025] Calculate the arithmetic mean of the three color channels corresponding to the target position based on the three color channel information;

[0026] Based on the arithmetic mean of the three color channels corresponding to the target position and the preset samples, the calibration parameters corresponding to the image sensor are calculated.

[0027] In a possible implementation, for each image sensor, the image sensor is further configured to dynamically adjust a frame period of a trigger signal corresponding to the image sensor according to a frame period of the image sensor and a preset period difference range;

[0028] The cycle difference range is determined based on the time for the sensor to collect a row of data and the number of output rows when blanking a row.

[0029] In a possible implementation manner, before the post-processing unit rearranges all the initial image sequences in time order to obtain the target image sequence, the post-processing unit is further used to perform image correction on all the initial image sequences according to the pre-acquired scene correction parameters to obtain all corrected initial image sequences;

[0030] Accordingly, the post-processing unit rearranges all the initial image sequences in chronological order to obtain a target image sequence, including:

[0031] The post-processing unit rearranges all corrected initial image sequences in time order to obtain a target image sequence.

[0032] In a possible implementation, the scene correction parameters include registration parameters and common area parameters, and the post-processing unit performs image correction on all initial image sequences according to the pre-acquired scene correction parameters to obtain all corrected initial image sequences, including:

[0033] The post-processing unit registers all the initial image sequences according to the registration parameters to obtain all the registered initial image sequences;

[0034] The post-processing unit selects and crops the common areas of all registered initial image sequences according to the common area parameters to obtain all corrected initial image sequences.

[0035] In a possible implementation manner, the N image sensors are arranged in a matrix arrangement or a spectroscopic arrangement, and the spectroscopic arrangement is a method of a spectroscopic prism plus the N image sensors.

[0036] In a second aspect, an embodiment of the present application provides a high frame rate imaging method, which is applied to the high frame rate camera described in any one of the first aspects, wherein the high frame rate camera includes an acquisition unit group and a processing unit group, wherein the acquisition unit group includes N image sensors, and the processing unit group includes M processing units and a post-processing unit, wherein N is an integer greater than or equal to 2, and M is an integer greater than or equal to 1; the method includes:

[0037] For each image sensor, in response to a user operation, the image sensor collects raw data based on a preset trigger signal, and transmits the raw data to a processing unit corresponding to the image sensor; wherein, for the first N-1 image sensors, the time difference between the trigger signals of adjacent image sensors is a first preset time, and the time difference between the trigger signals of the Nth image sensor and the N-1th image sensor is a second preset time, the first preset time is calculated based on the frame rate of each image sensor, and the second preset time is calculated based on the first preset time and the frame rate;

[0038] For each processing unit, the processing unit performs correction processing on the received raw data based on the corresponding ISP parameters to obtain an initial image sequence, and transmits it to the post-processing unit, wherein the ISP parameters are determined based on the calibration parameters of the image sensor corresponding to the raw data and the time difference between the trigger signal corresponding to the image sensor and the trigger signal corresponding to the previous image sensor;

[0039] The post-processing unit is used to rearrange all initial image sequences in time order to obtain a target image sequence.

[0040] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement possible implementation methods of the second aspect as above.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements a possible implementation method of the second aspect as above.

[0042] The high frame rate camera and imaging method provided by the embodiment of the present application include a collection unit group and a processing unit group, the collection unit group includes N image sensors, the processing unit group includes M processing units and a post-processing unit, wherein N is an integer greater than or equal to 2, and M is an integer greater than or equal to 1; for each image sensor, the image sensor is used to collect raw data based on a preset trigger signal and transmit the raw data to the processing unit corresponding to the image sensor; wherein, for the first N-1 image sensors, the time difference of the trigger signals of adjacent image sensors is a first preset time, and the time difference of the trigger signals of the Nth image sensor and the N-1th image sensor is a first preset time. The time difference is the second preset time, the first preset time is calculated based on the frame rate of each image sensor, and the second preset time is calculated based on the first preset time and the frame rate; for each processing unit, the processing unit is used to correct the received raw data based on the corresponding ISP parameters to obtain an initial image sequence, and transmit it to the post-processing unit, the ISP parameters are determined based on the calibration parameters of the image sensor corresponding to the raw data and the time difference between the corresponding trigger signal of the image sensor and the corresponding trigger signal of the previous image sensor, and the post-processing unit is used to rearrange all the initial image sequences in time order to obtain a target image sequence. Through precise time synchronization and data acquisition, accurate image correction processing and orderly image sequence reorganization, efficient and high-quality processing of raw data collected by multiple image sensors is achieved, thereby generating a high-quality target image sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] Figure 1 A schematic diagram of the system structure of a high frame rate camera provided in this application;

[0045] Figure 2 A schematic diagram of the data processing flow for the processing unit group;

[0046] Figure 3 This is a schematic diagram of the image sensor trigger signal control;

[0047] Figure 4 Schematic diagram of adjusting the external synchronization trigger signal period and the image sensor frame period;

[0048] Figure 5 It is a schematic diagram of the external synchronization trigger signal period and the image sensor frame period after adjustment;

[0049] Figure 6 It is a schematic diagram of the expansion of the number of acquisition frames;

[0050] Figure 7 Schematic diagram of matrix arrangement and combination;

[0051] Figure 8 Schematic diagram of the light splitting arrangement;

[0052] Fig. 9 It is a schematic diagram of data processing of the processing unit group;

[0053] Fig.10 Schematic diagram for adjusting white balance of a single image sensor;

[0054] Fig.11 Schematic diagram of exposure adjustment for a single image sensor;

[0055] Fig.12 Select a schematic diagram for the public area;

[0056] Fig.13 Rearrange the schematic diagram for transmission data flow;

[0057] Fig.14 A schematic diagram of the process flow of the high frame rate imaging method provided in this application.

[0058] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0059] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0060] A high frame rate camera is a camera that can capture high-speed dynamic scenes and can collect image data at a very high frequency, usually at a speed of tens to hundreds of frames per second, which is much higher than the frame rate of ordinary cameras. Due to the use of advanced sensors and image processing technology, high frame rate cameras can capture clearer images and more details. Due to the high frame rate, the amount of image data generated by high frame rate cameras in the same time is also larger, which puts higher requirements on storage and transmission capabilities. At present, the image sensor of high frame rate cameras needs to capture and transmit image data quickly, and high resolution means more data needs to be processed. When the resolution increases, the data output rate and processing power of the sensor become limiting factors. The simultaneous realization of high resolution and high frame rate requires the sensor to have higher sensitivity and faster response speed. However, these performance improvements are limited by material science, manufacturing process and other aspects. And once the hardware solution of the high frame rate camera is determined, its performance parameters such as resolution and frame rate are difficult to adjust significantly. This limits the flexibility and scalability of the camera. In summary, how to ensure the high frame rate of high frame rate cameras while improving scalability is a technical problem that needs to be solved urgently.

[0061] In response to the above problems, the present application provides a high frame rate camera and imaging method, which not only meets the needs of high frame rate applications, but also improves the scalability of the camera. Specifically, the image sensor of the high frame rate camera needs to capture and transmit image data quickly, and high resolution means that more data needs to be processed. When the resolution increases, the data output rate and processing power of the sensor become limiting factors. The simultaneous realization of high resolution and high frame rate requires the sensor to have higher sensitivity and faster response speed. However, these performance improvements are limited by material science, manufacturing process and other aspects. And once the hardware solution of the high frame rate camera is determined, its performance parameters such as resolution and frame rate are difficult to adjust significantly. This limits the flexibility and scalability of the camera. Taking these issues into consideration, the inventor studied whether it is possible to expand the frame rate by means of multiple image sensors to meet the needs of high frame rate applications and improve scalability. Based on this, the technical solution of this application is proposed.

[0062] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0063] Figure 1 The system structure diagram of the high frame rate camera provided in this application is as follows: Figure 1As shown, the system structure of the high frame rate camera includes an acquisition unit group and a processing unit group, wherein the acquisition unit group includes N image sensors, N≥2; the processing unit group includes M processing units and a post-processing unit, M≥1. The acquisition unit group, through synchronous control, sequentially acquires the image signals of the image sensor group. The acquired image data is sent to the processing unit group, which can be composed of a single processing element or multiple processing elements. Specifically:

[0064] For each image sensor, the image sensor is used to collect raw data based on a preset trigger signal and transmit the raw data to a processing unit corresponding to the image sensor, wherein, for the first N-1 image sensors, the time difference between the trigger signals of adjacent image sensors is a first preset time, and the time difference between the trigger signals of the Nth image sensor and the N-1th image sensor is a second preset time, the first preset time is calculated based on the frame rate of each image sensor, and the second preset time is calculated based on the first preset time and the frame rate.

[0065] For each processing unit, the processing unit is used to perform correction processing on the received raw data based on the corresponding ISP parameters to obtain an initial image sequence and transmit it to the post-processing unit, wherein the ISP parameters are determined based on the calibration parameters of the image sensor corresponding to the raw data and the time difference between the corresponding trigger signal of the image sensor and the corresponding trigger signal of the previous image sensor.

[0066] The post-processing unit is used to rearrange all the initial image sequences in time order to obtain a target image sequence.

[0067] For example, Figure 2 A schematic diagram of the data flow for the processing unit group is shown as follows: Figure 2 As shown, assuming that the image data output by image sensor 1 at different times are I11, I12...I1N, the image data output by image sensor 2 at different times are I21, I22...I2N, and the image data output by image sensor H at different times are IH1, IH2...IHN. After the processing unit, the image sequence is arranged as I11, I21...IH1, I12, I22...IH2..., I1N, I2N...IHN. Assuming that the frame rate of the image sensor is A fps, the rearranged image sequence frames reach A*N fps. It can meet the application requirements of high frame rate.

[0068] Optionally, in high-speed shooting scenarios, the images captured by each sensor need to be consistent in time so that subsequent image processing and data analysis can accurately reflect the motion state of the object. If there is no time synchronization between sensors, there may be a time difference between the images captured by different sensors, resulting in inconsistent information in the image sequence. In addition, when multiple sensors work at the same time, if there is no synchronization control between them, it may cause misalignment or distortion between images. Especially when shooting high-speed moving objects, this misalignment or distortion will be more obvious, thus affecting the accuracy and usability of the image. Especially in scientific experiments, medical imaging and other fields, high frame rate cameras are often used for precise measurement and calculation. Synchronous control can ensure that each sensor captures the image of the object at the same time, thereby improving the precision and accuracy of the measurement. Therefore, a high frame rate camera using multiple image sensors requires an accurate synchronization control strategy to improve the accuracy and image quality of the high frame rate camera.

[0069] Specifically, each image sensor is matched with a trigger signal source. Here, trigger signal 1 corresponding to image sensor 1 is selected as the reference, and the trigger signal sources corresponding to the other image sensors have fixed phases in sequence, and each trigger signal node time corresponds to the frame period of the image sensor. At the same time, it is necessary to ensure that the difference between the trigger signal node frame period and the sensor frame period does not exceed a certain range. If it exceeds this range, it may cause problems such as data not being able to be plotted, missing lines in data output, or too short inter-frame blanking.

[0070] For example, Figure 3 This is a schematic diagram of the image sensor trigger signal control, such as Figure 3 As shown in the figure, assuming that the frame rate of the image sensor is A fps, each frame takes up 1 / A time in ms, and the period of each trigger signal is also 1 / A ms. The phase difference between trigger signals is related to the number of image sensors. If there are N image sensors, considering that the deviation time may not be divisible, the first N-1 trigger signals lag behind the previous one. Time, that is, is the first preset time, the Nth trigger signal lags behind the previous one Time, that is, is the second preset time. And can be expressed by the formula:

[0071]

[0072]

[0073] in, is a floor function, A is the frame rate of the image sensor, and N is the number of image sensors.

[0074] In one possible implementation, Figure 4The schematic diagram of the external synchronous trigger signal cycle and the image sensor frame cycle before adjustment is as follows: Figure 4 As shown, in actual applications, there may be a mismatch between the external synchronization trigger signal period and the frame period corresponding to the image sensor. In order to eliminate the mismatch between the external synchronization trigger signal period and the frame period of the image sensor, an adjustment mechanism inside the sensor can be used. Specifically: for each image sensor, the image sensor is also used to dynamically adjust the frame period of the trigger signal corresponding to the image sensor according to the frame period of the image sensor and a preset period difference range.

[0075] The period difference range is the above-mentioned requirement to ensure that the frame period of the trigger signal node and the frame period of the sensor do not differ by more than a certain range. This range is determined based on the time it takes the image sensor to collect a row of data and the number of rows output when blanking a row, that is:

[0076]

[0077]

[0078] in, Indicates the time it takes for the image sensor to collect a row of data. Indicates the number of output lines when blanking lines. is the period difference range, and A is the frame rate of the image sensor.

[0079] It should be noted that the above 7 is an empirical value and can also be adjusted in real time according to the actual application scenario, and the embodiment of the present application does not make any specific limitation.

[0080] Exemplarily, the frame period of the image sensor and the trigger signal node period need to meet the following conditions:

[0081] Assume that the sensor frame period is VPS, and the frame period of the first N-1 trigger signal nodes is (and Association), according to the above requirements, the first N-1 trigger signals must meet:

[0082]

[0083] Assume that the frame period of the Nth trigger signal node is (and Association), according to the above requirements, the Nth trigger signal must meet:

[0084]

[0085] The process of dynamically adjusting the frame period of the trigger signal corresponding to the image sensor includes:

[0086] The arrival time of the external synchronous trigger signal cycle and the frame period of the image sensor are detected in real time. If the two do not meet the above conditions, the frame period of the trigger signal node is dynamically adjusted.

[0087] For example, Figure 5 Schematic diagram of the external synchronization trigger signal period and the image sensor frame period after adjustment. After adjustment, the relationship between the external synchronization trigger signal frame period and the corresponding frame period of the image sensor is as follows: Figure 5 As shown, the difference between the two intervals is less than the required .

[0088] It should be noted that Figure 6 This is a schematic diagram of the expansion of the number of acquisition frames, such as Figure 6 As shown, the image sensor outputs a signal only when triggered. Broadly speaking, the number of frames collected by the acquisition module increases by N times, that is, N*A fps.

[0089] In one possible implementation, the image sensor group can have different distributions, such as using a matrix arrangement of multiple sensors or using a beam splitter prism plus multiple sensors. Figure 7 It is a schematic diagram of matrix arrangement and combination, such as Figure 7 As shown, there is no requirement on the number of row and column image sensors. Figure 8 The following is a schematic diagram of the light splitting arrangement. The light splitting prism plus multi-eye sensor is used. The light splitting device is used to split the light energy N times, and the timing of each sensor being collected is controlled to achieve a high frame rate. Figure 8 As shown, the pressure of subsequent image registration correction can be reduced to a certain extent, but the light energy entering the sensor is weakened, which is suitable for scenes with strong lighting.

[0090] Optionally, to improve the accuracy and image quality of high frame rate cameras, Fig. 9 It is a schematic diagram of processing unit group data processing, such as Fig. 9 As shown, the processing unit group also needs to perform accurate data processing on the original data collected by the collection unit group.

[0091] Specifically, the ISP parameters include a white balance adjustment value and an exposure adjustment value, and the processing unit performs correction processing on the received raw data based on the corresponding ISP parameters to obtain an initial image sequence, specifically including:

[0092] For each processing unit, the processing unit adjusts the original data based on the white balance adjustment value to obtain adjusted original data.

[0093] The white balance adjustment value is determined based on the calibration parameters of the image sensor corresponding to the original data and the time difference between a trigger signal corresponding to the image sensor and a trigger signal corresponding to the previous image sensor.

[0094] The processing unit adjusts the adjusted original data based on the exposure adjustment value to obtain an initial image sequence.

[0095] The exposure adjustment value is determined based on a time difference between a trigger signal corresponding to the image sensor and a trigger signal corresponding to a previous image sensor.

[0096] There will be differences in the images (image performance, including color, noise, clarity, etc.) when multiple image sensors are collecting images, so image correction is required to ensure that the collected images are consistent.

[0097] Before the processing unit adjusts the original data based on the white balance adjustment value, it is necessary to determine the white balance adjustment value. Specifically:

[0098] For each time period of each raw data, a white balance statistic value is determined based on the average values ​​of the three color channels of the raw data corresponding to the time period.

[0099] The white balance calculation value corresponding to the time period is determined according to the white balance statistics value and the white balance calculation value of the previous image sensor of the image sensor corresponding to the original data.

[0100] Based on the white balance calculation value corresponding to the time period and the calibration parameters of the image sensor corresponding to the raw data, a white balance adjustment value corresponding to the time period is determined.

[0101] Before the processing unit adjusts the adjusted original data based on the exposure adjustment value, the exposure adjustment value needs to be determined. Specifically:

[0102] For each time period of each raw data, an exposure statistic value is determined based on brightness information of the raw data corresponding to the time period.

[0103] The exposure calculation value corresponding to the time period is determined according to the exposure statistics and the exposure calculation value of the previous image sensing of the image sensor corresponding to the raw data.

[0104] The calculated exposure value corresponding to the time period is determined as the exposure adjustment value corresponding to the time period.

[0105] For example, Fig.10 Schematic diagram of white balance adjustment for a single image sensor. Fig.11The following is a schematic diagram of exposure adjustment for a single image sensor. In the first time period, image sensor 1 corresponds to trigger signal 1 (T0), and image sensor N corresponds to trigger signal N (T0+(N-2)△T+△T_pay). In the T0 period, the calculated value of image sensor 1 is calculated based on the statistical information of image sensor 1, applied to image sensor 1, and multiplied by the difference caused by individual differences (calibration parameters of image sensor 1). The calculated value of image sensor 2 is obtained based on the statistical information of image sensor 2 and the calculated value of image sensor 1, applied to image sensor 2, and multiplied by the difference caused by individual differences (calibration parameters of image sensor 2). Similarly, the calculated value of image sensor N is obtained based on the statistical information of image sensor N and the calculated value of image sensor N-1, applied to image sensor N, and multiplied by the difference caused by individual differences (calibration parameters of image sensor N). In the T1 period, the calculated value of image sensor 1 is calculated based on the statistical information of image sensor 1 and the calculated value of image sensor N in the previous T0 period, applied to image sensor 1, and multiplied by the difference caused by individual differences (calibration parameters of image sensor 1). This process is the process of white balance adjustment. Fig.10 As shown. The difference between the exposure adjustment process and the white balance adjustment process does not need to be multiplied by the difference caused by individual differences, such as Fig.11 shown.

[0106] Assume that the exposure statistics of image sensor 1 are sta[ae][T0], and the white balance statistics are sta[awb][T0]. Based on the statistics, the calculated exposure adjustment value is adj[ae][T0], and the white balance adjustment value is adj[awb][T0]. The exposure statistics of image sensor 2 are sta[ae][ T0+△T], and the white balance statistics are sta[awb][ T0+△T]. Then the calculated exposure values ​​adj[ae][ T0+△T] and white balance values ​​adj[awb][ T0+△T] of image sensor 2 can be expressed as:

[0107]

[0108] And so on:

[0109] The Nth sensor adjustment value within a time period is:

[0110]

[0111] The first image sensor adjustment value for the next time period is:

[0112]

[0113] The white balance and exposure parameters can be counted and calculated according to the above steps. The parameters of other ISP processing units should be consistent and can be transmitted to the parameter control unit through the UART port, and then the parameter control unit will send it to each image sensor, so as to ensure the consistency of ISP processing.

[0114] It should be noted that the exposure time of the image sensors should not overlap, that is, the exposure time should be less than the above Therefore, this restriction needs to be added in the exposure control.

[0115] Optionally, before determining the white balance adjustment value, it is also necessary to determine the calibration parameters of the image sensor, which specifically includes:

[0116] For each image sensor, a background gray image pre-captured under a fixed light source environment is divided into a plurality of blocks.

[0117] Select the middle P blocks as the target position, where P is an integer.

[0118] The three-color channel information corresponding to the target position is extracted.

[0119] The arithmetic mean of the three color channels corresponding to the target position is calculated based on the three color channel information.

[0120] Based on the arithmetic mean of the three color channels corresponding to the target position and the pre-set samples, the calibration parameters corresponding to the image sensor are calculated.

[0121] Exemplarily, data collection requires capturing background gray statistical values ​​under a fixed light source environment. Considering the difficulty of placing the device during sampling, only the statistical information at the center of the picture is selected when obtaining statistical information (the middle P blocks are taken). The data at the middle position is taken as the statistical information of R, G, and B, and the arithmetic mean of the R, G, and B channels is calculated according to the BayerPattern format. When performing offline calibration, it is necessary to select a golden sample, use the golden sample as a reference, and move the individual data closer to the golden sample, and store the calculated correction value in the storage medium on the image sensor. The reference value of the golden sample is ref (R / G / B three channels). Assuming that the average value obtained by the i-th image sensor is ,

[0122]

[0123] Then, the correction value is

[0124]

[0125] Correction value That is the calibration parameter of the i-th image sensor.

[0126] It should be noted that P can be 9, but is not limited to 9, and can be adaptively set according to actual scene requirements.

[0127] After the processing unit completes the image correction of the raw data, it is transmitted to the post-processing unit. The images captured by multiple image sensors will have different scenes (image acquisition content), so scene correction is also required to ensure the consistency of the captured images. Specifically: the post-processing unit is also used to perform image correction on all initial image sequences according to the pre-acquired scene correction parameters to obtain all corrected initial image sequences.

[0128] The scene correction parameters include registration parameters and common area parameters. The post-processing unit performs image correction on all initial image sequences according to the pre-acquired scene correction parameters to obtain all corrected initial image sequences, including:

[0129] The post-processing unit registers all the initial image sequences according to the registration parameters to obtain all the registered initial image sequences.

[0130] The post-processing unit selects and crops the common areas of all registered initial image sequences according to the common area parameters to obtain all corrected initial image sequences.

[0131] For example, in order to make the images output by multiple image sensors consistent, scene correction is required through image registration and common area cropping. The registration parameters are a 3x3 matrix as shown below: represents the registration parameters of the nth image, , is the coordinate of the nth image.

[0132]

[0133]

[0134] in, , are the image coordinates before and after correction, is the transformation matrix of offline calibration.

[0135] Fig.12 Select a schematic diagram for the public area, such as Fig.12 As shown, after registration, the common area of ​​each image is solved as the input image according to the upper left corner coordinates (x_s, y_s) and the lower right corner coordinates (x_e, y_e), where the common area is the minimum inscribed rectangle of each image.

[0136] For example, Fig.13 Rearrange the diagram for the transmission data flow, such as Fig.13As shown, since multiple image sensors are involved, the data needs to be rearranged offline, and the data streams can be obtained sequentially in the order of acquisition.

[0137] It should be noted that although the data is processed with the same ISP parameters, the noise reduction module cannot refer to different image sensor frames, so the noise processing in the moving area under poor lighting conditions is slightly affected. Therefore, it is possible to consider performing a time domain noise reduction process on the back-end processor after rearranging the image sequence. If historical frame information is required, the historical frame information of the current image sensor and the historical frame information of the previous image sensor need to be weighted. The formula is as follows:

[0138]

[0139] Fig.14 A schematic diagram of the high frame rate imaging method provided in this application, such as Fig.14 As shown, the method is applied to the high frame rate camera of any of the above embodiments, wherein the high frame rate camera includes an acquisition unit group and a processing unit group, the acquisition unit group includes N image sensors, the processing unit group includes M processing units and a post-processing unit, wherein N is an integer greater than or equal to 2, and M is an integer greater than or equal to 1; the method includes:

[0140] S1401: For each image sensor, in response to a user operation, the image sensor collects raw data based on a preset trigger signal, and transmits the raw data to a processing unit corresponding to the image sensor.

[0141] S1402: For each processing unit, the processing unit performs correction processing on the received raw data based on the corresponding ISP parameters to obtain an initial image sequence, and transmits it to the post-processing unit.

[0142] S1403: The post-processing unit is used to rearrange all the initial image sequences in time sequence to obtain a target image sequence.

[0143] The specific implementation method of the above steps is the same as that in the previous embodiment and will not be repeated here.

[0144] The high frame rate imaging method provided in the embodiment of the present application is that for each image sensor, in response to user operation, the image sensor collects raw data based on a preset trigger signal, and transmits the raw data to the processing unit corresponding to the image sensor. For each processing unit, the processing unit corrects the received raw data based on the corresponding ISP parameters to obtain an initial image sequence, and transmits it to the post-processing unit, which is used to rearrange all the initial image sequences in chronological order to obtain a target image sequence. This method realizes efficient and high-quality processing of the raw data collected by multiple image sensors through precise time synchronization and data acquisition, accurate image correction processing, and orderly image sequence reorganization, thereby generating a high-quality target image sequence.

[0145] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0146] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0147] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0148] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0149] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0150] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0152] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0153] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0154] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A high frame rate camera, characterized in that: include: A collection unit group and a processing unit group, wherein the collection unit group includes N image sensors, and the processing unit group includes M processing units and post-processing units, wherein N is an integer greater than or equal to 2, and M is an integer greater than or equal to 1; For each image sensor, the image sensor is used to collect raw data based on a preset trigger signal, and transmit the raw data to a processing unit corresponding to the image sensor; wherein, in each acquisition cycle, for the first N-1 image sensors among the N image sensors, the time difference between the trigger signals of adjacent image sensors is a first preset time, and the time difference between the trigger signals of the Nth image sensor and the N-1th image sensor is a second preset time, and the first preset time and the second preset time are calculated by the following formula: in, is the first preset time, is the floor function, A is the frame rate of the image sensor, N is the number of image sensors, is the second preset time; For each processing unit, the processing unit is used to perform correction processing on the received raw data based on the corresponding ISP parameters to obtain an initial image sequence, and transmit it to the post-processing unit; The post-processing unit is used to rearrange all the initial image sequences in time order to obtain a target image sequence; The ISP parameters include a white balance adjustment value and an exposure adjustment value, and the processing unit performs correction processing on the received raw data based on the corresponding ISP parameters to obtain an initial image sequence, including: For each processing unit, the processing unit adjusts the raw data based on the white balance adjustment value to obtain adjusted raw data, wherein the white balance adjustment value is determined based on a calibration parameter of an image sensor corresponding to the raw data, a white balance statistical value of the image sensor corresponding to the raw data, and a white balance calculated value of a previous image sensor of the image sensor corresponding to the raw data; The processing unit adjusts the adjusted raw data based on the exposure adjustment value to obtain the initial image sequence, wherein the exposure adjustment value is determined based on the exposure statistics of the image sensor corresponding to the raw data and the exposure calculation value of the previous image sensor of the image sensor corresponding to the raw data.

2. The camera according to claim 1, characterized in that Also includes: For each time period of each raw data, determine a white balance statistic based on the average values ​​of the three color channels of the raw data corresponding to the time period; Determine a white balance calculation value corresponding to the time period according to the white balance statistics and a white balance calculation value of a previous image sensor of the image sensor corresponding to the original data; A white balance adjustment value corresponding to the time period is determined based on the white balance calculation value corresponding to the time period and a calibration parameter of the image sensor corresponding to the raw data.

3. The camera according to claim 1, characterized in that Also includes: For each time period of each raw data, determining an exposure statistic value based on brightness information of the raw data corresponding to the time period; Determine the exposure calculation value corresponding to the time period according to the exposure statistics and the exposure calculation value of the previous image sensing of the image sensor corresponding to the raw data; The calculated exposure value corresponding to the time period is determined as the exposure adjustment value corresponding to the time period.

4. The camera according to claim 1, characterized in that Also includes: For each image sensor, a background gray image pre-taken under a fixed light source environment is divided into a plurality of blocks; Select the middle P blocks as the target position, where P is an integer; Extracting the three-color channel information corresponding to the target position; Calculate the arithmetic mean of the three color channels corresponding to the target position based on the three color channel information; Based on the arithmetic mean of the three color channels corresponding to the target position and the preset samples, the calibration parameters corresponding to the image sensor are calculated.

5. The camera according to claim 1, characterized in that Before the post-processing unit rearranges all the initial image sequences in time order to obtain the target image sequence, the post-processing unit is further used to perform picture correction on all the initial image sequences according to the pre-acquired scene correction parameters to obtain all corrected initial image sequences; Accordingly, the post-processing unit rearranges all the initial image sequences in chronological order to obtain a target image sequence, including: The post-processing unit rearranges all corrected initial image sequences in time order to obtain a target image sequence.

6. The camera according to claim 5, characterized in that The scene correction parameters include registration parameters and common area parameters. The post-processing unit performs picture correction on all initial image sequences according to the pre-acquired scene correction parameters to obtain all corrected initial image sequences, including: The post-processing unit registers all the initial image sequences according to the registration parameters to obtain all the registered initial image sequences; The post-processing unit selects and crops the common areas of all registered initial image sequences according to the common area parameters to obtain all corrected initial image sequences.

7. The camera according to claim 4, characterized in that The N image sensors are arranged in a matrix arrangement or a spectroscopic arrangement, and the spectroscopic arrangement is a method of a spectroscopic prism plus the N image sensors.

8. A high frame rate imaging method, characterized in that: The high frame rate camera according to any one of claims 1 to 7, wherein the high frame rate camera comprises an acquisition unit group and a processing unit group, wherein the acquisition unit group comprises N image sensors, and the processing unit group comprises M processing units and a post-processing unit, wherein N is an integer greater than or equal to 2, and M is an integer greater than or equal to 1; the method comprises: For each image sensor, in response to a user operation, the image sensor collects raw data based on a preset trigger signal, and transmits the raw data to a processing unit corresponding to the image sensor; wherein, for the first N-1 image sensors, the time difference between the trigger signals of adjacent image sensors is a first preset time, and the time difference between the trigger signals of the Nth image sensor and the N-1th image sensor is a second preset time, the first preset time is calculated based on the frame rate of each image sensor, and the second preset time is calculated based on the first preset time and the frame rate; For each processing unit, the processing unit performs correction processing on the received raw data based on the corresponding ISP parameters to obtain an initial image sequence, and transmits the initial image sequence to the post-processing unit; The post-processing unit is used to rearrange all initial image sequences in time order to obtain a target image sequence.

Images