Image processing method, camera and computer program product

Abstract

The invention discloses an image processing method, a camera and a computer program product. The camera is provided with an image processing chip and an image sensor, the image sensor supports a first mode for collecting a first image at a specified first frame rate and a second mode for outputting a second image, and the resolution of the first image is smaller than that of the second image; the method applied to the camera comprises the following steps: after an image processing chip and an image sensor are electrified, initializing the image processing chip and triggering the image sensor to work in a first mode; the image processing chip obtains image convergence statistical information based on data obtained when the image sensor works in the first mode within the specified time period, determines target working parameters related to quick start according to the image convergence statistical information, and then controls the camera to be quickly started based on the target working parameters. Therefore, the camera outputs the image meeting the quality requirement to the user. According to the scheme, the camera can be quickly started at low cost.

Description

Technical Field

[0001] This application belongs to the technical field of data processing, and particularly relates to an image processing method, a camera, and a computer program product. Background Art

[0002] For a low-power camera, due to the need to save power, most of the time the camera will be in a low-power sleep state, and at this time the image processing chip of the camera is powered off. When an external event (such as a button press or detecting someone approaching, etc.) occurs, the camera will power on and start the image processing chip, so as to collect, process image data and output it for user use, such as previewing image data, storing image data, and / or pushing messages, etc.

[0003] For the above usage scenario, in order to achieve the quick start of the camera and enable the camera to output image data to the user "both well and quickly", the following solution is generally adopted: a photosensitive device is set on the camera to quickly collect scene information; on the one hand, referring to this scene information, the camera can configure an initial value close to the algorithm convergence result for relevant parameters such as exposure and white balance, so as to achieve the goal of early convergence of the quick-start image; on the other hand, referring to this scene information, the day-night mode to be adopted by the camera can be determined. In this solution, since a photosensitive device needs to be added, the cost of the camera will increase. Summary of the Invention

[0004] This application provides an image processing method, a camera, and a computer program product, which can achieve the quick start of the camera at a relatively low cost.

[0005] In a first aspect, this application provides an image processing method, which is applied to a camera. The camera is provided with an image processing chip and an image sensor. Among them, the image sensor supports a first mode and a second mode. The first mode collects first images at a specified first frame rate, and the second mode outputs second images, and the resolution of the first images is less than the resolution of the second images; the image processing method includes:

[0006] After the image processing chip and the image sensor are powered on, initialize the image processing chip and trigger the image sensor to work in the first mode;

[0007] The image processing chip obtains image convergence statistical information based on the data obtained when the image sensor works in the first mode within a specified time period;

[0008] After the specified time period, the image processing chip determines target working parameters related to quick start according to the image convergence statistical information;

[0009] The image processing chip controls the camera to start quickly based on the target operating parameters, so that the camera outputs images that meet the quality requirements to the user.

[0010] In a second aspect, the present application provides a camera. The camera includes a memory, an image sensor, and an image processing chip. Both the image sensor and the image processing chip are equipped with processors. A computer program that can run on the processors is stored in the memory. When the processors execute the computer program, the steps of the method in the first aspect are implemented.

[0011] In a third aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the method in the first aspect are implemented.

[0012] In a fourth aspect, the present application provides a computer program product. The computer program product includes a computer program. When the computer program is executed by one or more processors, the steps of the method in the first aspect are implemented.

[0013] The beneficial effects of the present application compared with the prior art are as follows: The image sensor carried by the camera of the present application can support the acquisition of small images at a high frame rate. After the image processing chip and the image sensor are powered on, the image sensor first acquires small images at a high frame rate. Since the frame rate is high and the data volume of the small images is small, the small images can be quickly converged to the normal brightness in a short time, so that the image processing chip can quickly obtain the image convergence statistical information based on the small images, and thereby determine the target operating parameters related to the quick start. In this way, the camera can be quickly started according to the target operating parameters, and thus output images that meet the quality requirements to the user as soon as possible. In the above process, the camera does not need to carry a photosensitive device, and the quick start of the camera can be realized at a lower cost.

[0014] It can be understood that the beneficial effects of the second to fourth aspects can refer to the relevant descriptions in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 It is a schematic flowchart of the implementation of the image processing method provided by the embodiment of the present application;

[0017] Figure 2It is an example diagram of the first mapping relationship provided by an embodiment of the present application;

[0018] Figure 3 It is an example diagram of the timing interaction process of each component of the camera when the camera uses an image processing method for fast startup provided by an embodiment of the present application;

[0019] Figure 4 It is a schematic structural diagram of the camera provided by an embodiment of the present application. Detailed implementation manners

[0020] To make the objectives, features, and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the embodiments described below are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0021] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0022] It should also be understood that the terms used in this specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification of the present application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0023] It should be further understood that the term "and / or" used in this specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] As used in this specification and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if detecting [the described condition or event]" can be interpreted as meaning "once determined", "in response to determining", "once detecting [the described condition or event]", or "in response to detecting [the described condition or event]" according to the context.

[0025] In addition, in the description of this application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0026] Generally speaking, achieving the goal of "good and fast" image output when a low-power camera starts quickly is related to the following three tasks:

[0027] 1. Decide the day / night mode according to the scene brightness;

[0028] 2. Decide the exposure parameters to be configured according to the scene brightness so that the image exposure is appropriate;

[0029] 3. Decide the white balance parameters to be configured according to the scene color temperature so that the image color is normal.

[0030] Among them, the above task 2 is usually implemented by the automatic exposure (AE) algorithm of the image processing chip, and task 3 is usually implemented by the automatic white balance (AWB) algorithm of the image processing chip. In actual applications, the algorithm usually requires multiple frames of images to gradually converge the corresponding parameters; especially in special scenarios such as high color temperature scenes, low color temperature scenes, high illumination scenes, and low illumination scenes, the algorithm convergence often takes longer. Based on this, the current optimization solution is to quickly measure the scene information; referring to this scene information, the camera can configure an initial value close to the algorithm convergence result for relevant parameters such as exposure and white balance, so as to achieve the goal of early convergence of the quick-start image; in addition, referring to this scene information, the camera can also decide the day / night mode to be adopted by the camera.

[0031] Currently, most low-power cameras use photosensitive devices to collect scene information, so as to assist the image processing chip to complete the convergence work of the quick-start image. It can be understood that the photosensitive sensor can convert the scene brightness information into voltage. In this way, after the image processing chip collects the voltage value through the analog-to-digital converter (ADC), it is equivalent to obtaining the scene brightness value, and then it can assist the image processing chip to complete tasks 1 and 2 of the quick-start image convergence. The specific process is briefly described as follows:

[0032] Task 1: The camera can pre-determine a threshold for day-night mode switching. For example, in a laboratory scenario with a light intensity of 2 lux, if the photosensitive reading value read by the ADC is 15, then this value 15 can be determined as the threshold. In an actual scenario, once the photosensitive reading value exceeds this threshold, it can be considered that the current scene is bright, and the daytime mode can be adopted, that is, turning off the infrared fill light and / or switching the built-in dual filter (IR-CUT) to a red filter; conversely, once the photosensitive reading value does not exceed this threshold, it can be considered that the current scene is dark, and the night vision mode can be adopted, that is, turning on the infrared fill light and / or switching the IRCUT to a white filter.

[0033] Task 2: Construct various brightness scenes in advance and calibrate the optimal exposure parameters (including night vision mode) for each scene's brightness. In an actual scenario, based on the photosensitive reading value, interpolation calculations can be performed among the calibrated exposure parameters of each gear, so as to calculate the initial exposure value.

[0034] However, the above scheme using photosensitive devices still has the following problems:

[0035] Problem 1: It has high requirements for the accuracy and range of photosensitive devices.

[0036] According to the previous description, it is necessary to set the divided values of each calibrated scene based on the photosensitive reading value. Then the two most basic requirements are: 1) High discrimination for changes in scene brightness; 2) Comprehensive coverage of the range of scene brightness changes. However, these two requirements are contradictory in themselves. Therefore, during circuit debugging, only a balance can be achieved as much as possible.

[0037] For example, in order to meet the requirement of high discrimination, the threshold for day-night switching is set to 2 lux, that is, in a scene with a brightness exceeding 2 lux, the daytime mode is adopted, otherwise the night vision mode is switched; then, in a 2-lux scene, the circuit needs to be debugged to make the photosensitive reading value exceed 15, so that several brightness gears with relatively high contrast can be divided within the low-light night vision brightness range (such as 5 gears), and then the initial values of different exposure parameters for each gear can be calibrated. And in an actual scenario, there will not be an easy cross-gear situation due to a certain tolerance difference in the photosensitive reading values of each gear. However, the above debugging will necessarily reduce the photosensitive range, resulting in the loss of discrimination due to the photosensitive reading value reaching the maximum in a high-brightness scene. For example, the photosensitive reading values are saturated both indoors in high brightness (such as 100 lux) and outdoors in high brightness (such as 1000 lux). At this time, since the scene brightness gears cannot be distinguished from the read values, only the initial value of the same gear of exposure parameters can be configured; if this gear of parameters happens to make the exposure appropriate in the indoor high-brightness scene, then using the same exposure parameter initial value in the outdoor high-brightness scene will necessarily cause overexposure of the picture.

[0038] Therefore, the above scheme using photosensitive devices has high requirements for the accuracy and range of photosensitive devices, which will lead to an increase in the cost of photosensitive devices.

[0039] Problem 2: There are relatively high requirements for the consistency of photosensitive devices.

[0040] As can be seen from the introduction to Problem 1 above, if a photosensitive device is to ensure coverage of the scene brightness range, the calibration gradings within the low-light night vision range must have relatively small differences. If the consistency of the photosensitive device is very poor, it is very likely that the actual performance of the factory products will deviate significantly from the calibration. For example, a median photosensitive device is used to calibrate the night vision gear, and the initial values of three sets of parameters are calibrated under the scenes with photosensitive readings of 12, 9, and 6, namely P-12, P-9, and P-6 respectively. Assume that the reading of a deviated photosensitive device is on the low side. Then, in the scene where the reading of the median photosensitive device is 12, the reading of the deviated photosensitive device is exactly 6, resulting in the camera using the initial value of the parameters calibrated for the P-6 gear to perform the first-frame exposure. Since the P-6 gear has deviated from the P-12 gear by 2 gears, the initial value of the parameters used must be inappropriate, leading to abnormal first-frame exposure.

[0041] Moreover, for the threshold used to decide the day-night mode switch, it is also affected by the consistency of the photosensitive device, which may lead to inconsistent decisions obtained based on different photosensitive devices in the same scene.

[0042] Problem 3: Photosensitive devices cannot provide the color temperature information of the scene.

[0043] Photosensitive devices cannot provide the color temperature information of the scene, and they have no reference value for the initial value of the white balance gain of the image. That is to say, the above scheme using photosensitive devices cannot assist the image processing chip to complete Task 3 described above. Therefore, the white balance convergence of the image can only be handed over to the AWB algorithm of the image processing chip. However, in some low-color-temperature or high-color-temperature scenes, the AWB algorithm usually requires multiple-frame processing to converge. If high-quality first-frame images without color cast are to be provided to users, frames must be appropriately discarded, which will undoubtedly lead to a delay in the first-frame time.

[0044] Problem 4: It takes time for the photosensitive reading to stabilize.

[0045] After power-on, the photosensitive reading usually has a convergence process and often takes hundreds of milliseconds to stabilize. A reasonable approach is to use the stabilized photosensitive reading for scene brightness calibration. However, for a quick startup, the photosensitive reading is urgently needed for tasks such as day / night mode decision-making and initial value calculation, and it is impossible to wait for too long a convergence time. Based on this, only the transient value can be used to calibrate the scene brightness. However, the prerequisite for using the transient value is that the image processing chip must obtain the photosensitive reading at a fixed time each time. Otherwise, the photosensitive reading cannot correctly reflect the scene brightness. This places high requirements on the running time of the image processing chip. Once software changes affect the timing of obtaining the photosensitive reading, or replacing the image processing chip solution causes a change in the timing of obtaining the photosensitive reading, all calibration results will be invalid.

[0046] In addition, to prevent abnormal fluctuations in the photosensitive reading, it is usually necessary to take the average or median value after multiple readings for determination, which also slows down the quick startup speed to a certain extent.

[0047] Based on this, the embodiments of the present application propose an image processing method, a camera, and a computer program product, which no longer rely on photosensitive devices to achieve a quick startup, thereby eliminating the various problems mentioned above and enabling the quick startup of the camera at a lower cost.

[0048] Next, the image processing method provided by the embodiments of the present application will be described. Among them, the image processing method is applied to a camera, and the camera is provided with an image processing chip and an image sensor. The image sensor used in the embodiments of the present application supports two working modes, namely the first mode and the second mode.

[0049] Specifically, the first mode, also known as the small picture mode. In this first mode, the image sensor can collect a first image at a specified first frame rate. Among them, the first image is a small picture, that is, the image sensor outputs an image in the binning mode, and the output resolution is much lower than the normal output resolution; and the first frame rate is a relatively high frame rate. That is, in this first mode, the image sensor can collect small pictures at a high frame rate.

[0050] Specifically, the second mode, also known as the normal mode. In this second mode, the image sensor can output a second image with a normal resolution. As described above, the resolution of the second image is greater than that of the first image; and the output frame rate in the second mode can be adjusted according to the actual application scenario, but its default output frame rate is generally much lower than the first frame rate.

[0051] Based on the first mode, the image sensor can have a built-in small core to support the Fast Auto Exposure (FastAE) feature, which specifically means that: after the image sensor automatically controls the exposure and acquires a small image (i.e., small-sized frame data) in binning mode, instead of transmitting the small image to the outside, the built-in small core of the image sensor analyzes the small image (including but not limited to statistical analysis of the average luminance information of the small image, etc.), and automatically adjusts the exposure parameters of the image sensor itself (including but not limited to exposure time, analog gain, and digital gain, etc.); then, the image sensor acquires a new frame of small image according to the latest exposure parameters and repeats the above steps to complete the rapid convergence of the exposure parameters based on the small image in the current scene.

[0052] Among them, the process of the image sensor adjusting its own exposure parameters varies according to the manufacturer of the image sensor, but the general process can be briefly described as: after exposure and processing according to the initial exposure parameters, a small image is obtained; then, the average luminance of the small image is statistically analyzed, and the obtained current average luminance is compared with the preset target luminance of the small image; if the current average luminance is lower / higher than the target luminance of the small image, the exposure parameters are appropriately increased / decreased.

[0053] It can be understood that through the above process, the current average luminance of the obtained small image can approach the target luminance of the small image frame by frame. After finally reaching the convergence threshold range, it can be considered that the exposure converges (i.e., the rapid convergence of the exposure parameters based on the small image is achieved).

[0054] Based on the above description of the camera proposed in the embodiments of the present application, please refer to Figure 1 , the image processing method proposed in the embodiments of the present application includes:

[0055] Step 101, after the image processing chip and the image sensor are powered on, initialize the image processing chip and trigger the image sensor to work in the first mode.

[0056] After the image processing chip and the image sensor are powered on, for the image processing chip, it will first perform an initialization operation; for the image sensor, it will default to start working in the first mode. Among them, the image processing chip and the image sensor may be powered on because the camera is awakened from a low-power or standby state, and the power-on timing of the image processing chip and the image sensor is not limited here.

[0057] It should be noted that after the image sensor is powered on and started, an MCLK clock signal needs to be provided by the outside. In the embodiments of the present application, this clock signal can specifically be provided by the image processing chip; that is, in the embodiments of the present application, it can be required that the image processing chip provides this clock signal as early as possible after the image sensor is powered on, so that the image sensor can start working in the first mode as early as possible.

[0058] Step 102: The image processing chip obtains image convergence statistical information based on the data obtained when the image sensor operates in the first mode within a specified time period.

[0059] The specified time period refers to the time period from when the image sensor starts to operate in the first mode until the exposure of the first image has converged (i.e., the small image exposure has converged). It can be understood that due to the characteristics of the first image having a lower resolution and a higher frame rate, the first image can quickly reach exposure convergence, so that the image processing chip can obtain the image convergence statistical information as soon as possible; that is, generally speaking, this specified time period is relatively short.

[0060] Based on the three works related to fast startup proposed above (including Work 1, Work 2, and Work 3), the image convergence statistical information in the embodiments of this application includes, but is not limited to: exposure statistical information and white balance gain statistical information.

[0061] Among them, the exposure statistical information can be obtained by statistically analyzing the first image whose exposure has converged after the exposure of the first image has converged. Specifically, it can include, but is not limited to, the following data: small image statistical brightness, small image exposure time, and small image exposure gain.

[0062] Among them, the white balance gain statistical information can be calculated by statistically analyzing the mean values of pixels R / G and B / G in the frame of the first image when the image sensor operates in the first mode. Specifically, it can include, but is not limited to, the following data: R pixel gain statistical data and B pixel gain statistical data. For the convenience of description, hereinafter, the R pixel gain statistical data will be denoted as RGain, and the B pixel gain statistical data will be denoted as BGain.

[0063] In some application scenarios, the image sensor has a small core built-in and supports the FastAE feature. Based on the description of the FastAE feature above, it can be known that when the image sensor operates in the first mode, it is specifically in a high-speed convergence mode of "output small image -> statistics & calculation -> adjust exposure parameters -> output small image -> statistics & calculation -> adjust exposure parameters ->...". It can be understood that in this application scenario, the image sensor can autonomously statistically obtain the required image convergence statistical information and store the image convergence statistical information in a plurality of preset registers; that is, the data obtained when the image sensor operates in the first mode already includes the image convergence statistical information, and it does not need to transmit the small image (i.e., the first image) to the outside (such as the image processing chip). Since the exposure convergence speed of the small image is very fast, the image processing chip can, after initialization is completed, read each register of the image sensor through I2C (Inter-Integrated Circuit) to obtain the image convergence statistical information.

[0064] In some other application scenarios, the image sensor does not have a small core built-in and does not support the FastAE feature. In this case, the image sensor can transmit the first image to the image processing chip. The image processing chip performs statistical and calculation operations and adjusts the exposure parameters of the image sensor until exposure convergence is achieved. Finally, the image processing chip obtains the first image with exposure convergence and obtains image convergence statistical information through statistical analysis. Compared with the previous application scenario, in this application scenario, the data obtained when the image sensor operates in the first mode only includes the first image, and some operations are transferred to the image processing chip for execution. Since this process involves the transmission of the first image and chip calculations, etc., the time consumption increases; that is, compared with the previous application scenario, the speed of the camera's fast startup decreases in this application scenario.

[0065] Step 103, after a specified time period, the image processing chip determines target working parameters related to fast startup according to the image convergence statistical information.

[0066] Based on the work 1, work 2, and work 3 related to fast startup described above, it can be known that the target working parameters to be determined by the image processing chip may include one or more of the following: the scene brightness related to work 1, the initial value of the exposure parameter related to work 2, and the initial value of the white balance gain parameter related to work 3. The determination methods of each target working parameter are specifically described below:

[0067] Regarding the scene brightness related to work 1, it is obtained depending on the exposure statistical data, and the details are as follows:

[0068] The final brightness during image sensor imaging is related to the scene brightness and exposure parameters. Specifically, when the exposure parameters remain unchanged, the higher the scene brightness, the brighter the final brightness during imaging; when the scene brightness remains unchanged, the larger the exposure parameters (that is, the longer the exposure time and the larger the exposure gain), the brighter the final brightness during imaging. Based on this, the formula for image brightness can be specifically as follows:

[0069] Image brightness = k × scene brightness × exposure time × exposure gain

[0070] Among them, k is a proportionality coefficient. Based on the above formula for image brightness, the formula for scene brightness can be inversely deduced, and it can be specifically as follows:

[0071] Scene brightness = 1 / k × [image brightness / (exposure time × exposure gain)]

[0072] Therefore, in the embodiments of the present application, the exposure statistical data can be substituted into the above formula, enabling the image processing chip to calculate the scene brightness based on the exposure statistical data and the preset proportionality coefficient k. That is, the image processing chip can substitute the small image statistical brightness into the image brightness in the above formula, substitute the small image exposure time into the exposure time in the above formula, and substitute the small image exposure gain into the exposure gain in the above formula, so as to calculate the scene brightness. Among them, the proportionality coefficient k can be determined according to actual measurements and is not limited here.

[0073] Regarding the initial values of the exposure parameters related to Work 2, they are also obtained depending on the exposure statistical data, and are described in detail as follows:

[0074] The image sensor can first obtain the scene brightness. This scene brightness can be specifically obtained through the exposure statistical data, and its acquisition method has been described above and will not be elaborated here. On this basis, R & D personnel can pre-construct multiple different brightness levels and calibrate the most suitable exposure parameters for each brightness level, so as to construct the brightness-exposure parameter correspondence relationship. In this way, the image processing chip can determine the initial values of the exposure parameters by means of linear interpolation calculation based on the obtained scene brightness and this brightness-exposure parameter correspondence relationship.

[0075] Of course, the above method can also be applied only in the night vision scene. That is, multiple brightness levels are pre-constructed and calibrated only in the night vision scene; correspondingly, subsequently, only when the scene brightness is less than the preset brightness threshold, the image processing chip determines the initial values of the exposure parameters according to the scene brightness and the preset brightness-exposure parameter correspondence relationship. For the daytime scene, the image processing chip can directly calculate the initial values of the exposure parameters. It can be understood that since the scene where the image sensor outputs images in the first mode currently is the same as the scene where it outputs images in the second mode subsequently, the image processing chip can directly adjust the allocation of the exposure parameters according to the target brightness when outputting images normally (that is, outputting images in the second mode). The above process can be expressed by the following formula:

[0076] In the scene of outputting images in the first mode, it can be obtained that:

[0077] Scene brightness = k0 × small image statistical brightness / (small image exposure time × small image exposure gain)

[0078] In the scene of outputting images in the second mode, it can be obtained that:

[0079] Scene brightness = k1 × target brightness / (output image exposure time × output image exposure gain)

[0080] Since the scenes are the same (equivalent to the same scene brightness), the following formula can be obtained by transformation:

[0081] Output image exposure amount = k2 × (target brightness / small image statistical brightness) × small image exposure amount

[0082] Among them, the exposure amount for image output is the image output exposure time × the image output exposure gain; the exposure amount for thumbnail is the thumbnail exposure time × the thumbnail exposure gain. k2 is a preset proportionality coefficient, which can be determined according to actual measurement. For example, it can be set to 3.75% etc., and the value is not limited here. Thus, after calculating the exposure amount for image output, the calculated exposure amount for image output can be reasonably allocated to the exposure time and exposure gain according to the image output frame rate and trailing requirements in the scenario of image output in the second mode, so as to obtain the initial value of the exposure parameters.

[0083] Regarding the initial value of the white balance gain parameter related to Job 3, it is obtained depending on the white balance gain statistical data, which is described in detail as follows:

[0084] The image processing chip determines the initial value of the white balance gain parameter according to the white balance gain statistical data. As described above, the image convergence statistical data are all stored in the registers of the image sensor; specifically for the white balance gain statistical data, the RGain and BGain included therein can be respectively stored in the corresponding preset registers. Since the number of bits of the register is limited, the result stored therein may not accurately represent the white balance gain. For example, at extremely high color temperatures, the R component in the exposed and converged thumbnail is very small, so RGain will be very large, which may exceed the expression range of the register corresponding to RGain, but the value of the register corresponding to BGain is still of reference value at this time. For this reason, the embodiment of the present application can pre-construct a white balance parameter mapping relationship, which includes: a first mapping relationship, and a second mapping relationship. Among them, the first mapping relationship is used to express the mapping relationship between the read value data pair of RGain-BGain and the expected value data pair when RGain has reached the maximum value of the register; the second mapping relationship is used to express the mapping relationship between the read value data pair of RGain-BGain and the expected value data pair when BGain has reached the maximum value of the register.

[0085] Please refer to Figure 2 , Figure 2 which gives an example diagram of the first mapping relationship. Among them, A, B, C, and D represent four different scenarios, 0 in the brackets represents the read value of the register, and 1 in the brackets represents the expected value; that is, A(0) represents the read value data pair of RGain-BGain obtained through the register in a certain scenario, and A(1) represents the expected data pair of RGain-BGain in this scenario (that is, in this scenario, the RGain and BGain finally converged by the image processing chip through its AWB algorithm for processing the normal image transmitted by the image sensor).

[0086] Based on this, when the R pixel gain statistical data or the B pixel gain statistical data has reached the maximum value of the corresponding register, the image processing chip can determine the initial value of the white balance gain parameter by linear interpolation according to the preset white balance parameter mapping relationship and the white balance gain statistical data.

[0087] As Figure 2 shown, assume that A(0) is (2040, 1480), B(0) is (2040, 1560), A(1) is (2364, 1442), B(1) is (2082, 1552), and assume that after the exposure of the small picture converges, the read value data pair X(0) of RGain - BGain is (2040, 1500). Then, according to the relative position of this X(0) on the line segment A(0)B(0), it can be mapped to the corresponding position X(1) point on the line segment A(1)B(1), that is, (2293, 1469).

[0088] Conversely, when neither the R pixel gain statistical data nor the B pixel gain statistical data has reached the maximum value of the corresponding register, the image processing chip can directly determine the white balance gain statistical data as the initial value of the white balance gain parameter.

[0089] Step 104, the image processing chip controls the camera to start quickly based on the target working parameters, so that the camera outputs an image that meets the quality requirements to the user.

[0090] The image processing chip can control the camera to start quickly based on the target working parameters, specifically: making a switching decision between day and night modes based on the scene brightness; configuring the image sensor based on the initial value of the exposure parameter so that the exposure of the first second image (i.e., the normal image) output by the image sensor is appropriate; and processing the first second image (i.e., the normal image) transmitted by the image sensor based on the initial value of the white balance gain so that the color of the first second image (i.e., the normal image) is normal.

[0091] Specifically, for operation 1, when the obtained scene brightness is less than the preset brightness threshold, the night vision mode can be switched, so that the image processing chip triggers the infrared fill light to turn on; of course, as described above, there can also be other operations such as switching the IR - CUT to a white sheet, and the operations that can be performed in the night vision mode are not limited here.

[0092] Specifically, for operation 2, the image processing chip can set an initialization sequence for configuring the image sensor based on the initial value of the exposure parameter, and send the initialization sequence to the image sensor. It can be understood that after receiving the initialization sequence, the image sensor can perform configuration operations based on the initialization sequence and switch its working mode from the first mode to the second mode. In this way, the image sensor can perform exposure and image output based on the initial value of the exposure parameter, obtain the first second image and transmit it to the image processing chip. The image processing chip can run the AE algorithm based on the first second image, thereby obtaining a new exposure parameter and transmitting it to the image sensor, so that the image sensor performs exposure and image output based on the new exposure parameter, obtains the second second image and transmits it to the image processing chip. By analogy, the exposure of the second image is finally converged, which will not be elaborated here.

[0093] Specifically, for operation 3, when the image processing chip receives the first second image transmitted by the image sensor, it can adjust the color of the first second image based on the initial value of the white balance gain parameter to make the color of the first second image normal; at the same time, the image processing chip can also run the AWB algorithm on the first second image before color adjustment, thereby obtaining a new white balance gain parameter. After that, when the image processing chip receives the second second image transmitted by the image sensor, it can adjust the color of the second second image based on the new white balance gain parameter to make the color of the second second image normal; in addition, the image processing chip can also run the AWB algorithm on the second second image before color adjustment, thereby obtaining a new white balance gain parameter again. By analogy, the white balance of the second image is finally converged, which will not be elaborated here.

[0094] In some embodiments, in most application scenarios, the processed first-frame second image obtained based on the initial values of the exposure parameters and the initial values of the white balance gain parameters obtained above can meet the quality requirements. However, in extreme application scenarios, it is still possible that the initial values of the exposure parameters and / or the initial values of the white balance gain parameters are not accurate enough, resulting in inappropriate exposure and / or white balance. Based on this, to further ensure the quality of the second image output to the user, in the embodiments of the present application, after the image processing chip processes the second image output by the image sensor, if the processed second image does not meet the preset conditions, the processed second image can be discarded until the processed second image meets the preset conditions, and then the processed second image starts to be output to the user. That is, the image processing chip can discard the first several frames of images according to the actual situation, and use the second image obtained after performing the AWB / AE algorithm for several rounds as the first frame output to the user. In some examples, the preset condition may be that the processed second image meets the quality requirements. In other examples, the preset condition may also be that the total number of processed second images (including those already discarded) has reached a preset quantity threshold N, where N is an integer greater than or equal to 0; it can be understood that when N is 0, the image processing chip can directly output the first processed second image as the first frame to the user; when N is an integer greater than 0, the image processing chip will actually output the Nth processed second image as the first frame to the user, that is, the first to the N-1th processed second images are all discarded. Of course, the preset conditions can also be set according to other requirements, and the embodiments of the present application do not limit the preset conditions.

[0095] In this case, to make the second image output to the user fast enough, in the embodiments of the present application, after the image sensor switches to work in the second mode, the image output frame rate of the image sensor can be set to the second frame rate first, so that the image sensor outputs the second image at the specified second frame rate first. In this way, the delay caused by frame dropping will be reduced. After the image processing chip starts to output the processed second image to the user (that is, after the processed second image meets the preset conditions), the image processing chip can control the image output frame rate of the image sensor to be restored from the second frame rate to the default third frame rate, so that the image sensor outputs subsequent images at the third frame rate. Obviously, the third frame rate is less than the second frame rate. That is, the image sensor outputs images at a high frame rate in the initial stage of switching to the second mode, and then resumes outputting images at the normal default frame rate.

[0096] Please refer to Figure 3 , Figure 3 , which gives an example of the timing interaction process of each component of the camera (including the image sensor, the image processing chip, the infrared fill light, and the IR-CUT of the lens) when the camera uses the image processing method proposed in the examples of the present application for quick startup. Among them,Figure 3 The image sensor shown supports the FastAE feature. The following combines with Figure 3 to describe this interaction process.

[0097] After the camera is powered on, the image processing chip is initialized first, so as to provide the MCLK clock signal for the image sensor, enabling the image sensor to start working in the first mode. In this first mode, the image sensor can quickly achieve the exposure convergence of the small image, and the image convergence statistical information obtained after convergence can be stored in the register.

[0098] The image processing chip reads the register to obtain the image convergence statistical information, and calculates the target working parameters based on this image convergence statistical information, including the scene brightness, the initial values of the exposure parameters, and the initial values of the white balance gain parameters.

[0099] The image processing chip can make a decision on the day / night mode based on the scene brightness to determine whether to turn on the infrared fill light and which color to switch the IR-CUT to.

[0100] The image processing chip can also send an initialization sequence to the image sensor based on the initial values of the exposure parameters, so as to set the exposure of the image sensor and at the same time trigger the image sensor to work in the second mode. The image sensor collects the first second image based on the currently set exposure (set based on the initial values of the exposure parameters) and transmits it to the image processing chip. For this first second image, on the one hand, the image processing chip adjusts its color through the initial values of the white balance gain parameters, and the color-adjusted first second image can be output to the display screen for the user to view; on the other hand, the image processing chip applies the AE algorithm and the AWB algorithm to it to obtain new exposure parameters and new white balance gain parameters.

[0101] Subsequently, the image processing chip continues to set the exposure of the image sensor based on the new exposure parameters. The image sensor collects the second second image based on the currently set exposure (set based on the new exposure parameters) and transmits it to the image processing chip. For this second second image, on the one hand, the image processing chip adjusts its color through the new white balance gain parameters, and the color-adjusted second second image can be output to the display screen for the user to view; on the other hand, the image processing chip applies the AE algorithm and the AWB algorithm to it to obtain new exposure parameters and new white balance gain parameters again.

[0102] Next, the image processing chip continues to set the exposure of the image sensor based on the new exposure parameters. The image sensor acquires the third-frame second image based on the currently set exposure (set based on the new exposure parameters) and transmits it to the image processing chip. For this third-frame second image, on the one hand, the image processing chip adjusts its color using the new white balance gain parameters, and the color-adjusted third-frame second image can be output to the display screen for the user to view; on the other hand, the image processing chip applies the AE algorithm and the AWB algorithm to it, thereby obtaining new exposure parameters and new white balance gain parameters again.

[0103] And so on until both the white balance gain parameters and the exposure parameters reach convergence, which will not be elaborated here.

[0104] It can be understood that in general application scenarios, the difference between the converged white balance gain parameters and the initial values of the white balance gain parameters is not too large, and the difference between the converged exposure parameters and the initial values of the exposure parameters is not too large either. Therefore, it can ensure that the second image output for the user to view can meet the quality requirements, achieving the goal of "good and fast" image output when the camera starts up quickly. Of course, in extreme application scenarios, the first N - 1 frames of the color-adjusted second images can be directly discarded instead of being output to the user for viewing, and the frame rate of the image output by the image sensor before the Nth frame can be adjusted to a high frame rate, with other operations remaining unchanged. For specific details, please refer to the previous description and will not be elaborated here.

[0105] As can be seen from the above, in the embodiment of the present application, the image sensor carried by the camera can support outputting small images at a relatively high frame rate; after the camera is powered on, the image sensor first outputs small images at a relatively high frame rate. Since the frame rate is high and the data volume of the small images is small, the small images can quickly converge to normal brightness in a short time, enabling the image processing chip to quickly obtain image convergence statistical information based on the small images and thereby determine the target working parameters related to quick startup. In this way, the camera can quickly start up according to the target working parameters and thus output images that meet the quality requirements to the user as soon as possible. In the above process, the camera does not need to carry photosensitive devices, and can achieve the quick startup of the camera at a relatively low cost.

[0106] Corresponding to the image processing method provided above, the embodiment of the present application also provides a camera. Please refer to Figure 4 , the camera 4 in the embodiment of the present application includes: a memory 401, an image sensor, and an image processing chip (in Figure 4(not shown in the figure). Among them, the image sensor supports a first mode and a second mode. The first mode acquires first images at a specified first frame rate, and the second mode outputs second images, and the resolution of the first images is less than the resolution of the second images. Both the image sensor and the image processing chip are equipped with a processor 402, and a computer program that can run on the processor 402 is stored in the memory 401. Specifically, when the processor 402 runs the computer program stored in the memory 401, the following steps are implemented:

[0107] After the image processing chip and the image sensor are powered on, initialize the image processing chip and trigger the image sensor to work in the first mode;

[0108] The image processing chip obtains image convergence statistical information based on the data obtained when the image sensor works in the first mode within a specified time period;

[0109] After the specified time period, the image processing chip determines target working parameters related to fast startup according to the image convergence statistical information;

[0110] The image processing chip controls the camera to start quickly based on the target working parameters, so that the camera outputs images that meet the quality requirements to the user.

[0111] Assume that the above is the first possible implementation manner. Then, in the second possible implementation manner provided based on the first possible implementation manner, the image convergence statistical information includes: exposure statistical data; the target working parameters include: scene brightness; the image processing chip determines the target working parameters related to fast startup according to the image convergence statistical information, including:

[0112] The image processing chip calculates the scene brightness according to the exposure statistical data and a preset proportional coefficient.

[0113] In the third possible implementation manner provided based on the second possible implementation manner above, the camera is also provided with an infrared fill light; the image processing chip controls the camera to start quickly based on the target working parameters, including:

[0114] In the case where the scene brightness is less than a preset brightness threshold, the image processing chip triggers the infrared fill light to turn on.

[0115] In the fourth possible implementation manner provided based on the second possible implementation manner above, the target working parameters further include: the initial value of the exposure parameter; the image processing chip determines the target working parameters related to fast startup according to the image convergence statistical information, and further includes:

[0116] In the case where the scene brightness is less than a preset brightness threshold, the image processing chip determines the initial value of the exposure parameter according to the scene brightness and a preset brightness-exposure parameter correspondence.

[0117] In the fifth possible implementation provided based on the above second possible implementation, the target operating parameters further include: the initial value of the exposure parameter; the image processing chip determines the target operating parameters related to fast startup according to the image convergence statistical information, and further includes:

[0118] When the scene brightness is greater than the preset brightness threshold, the image processing chip determines the initial value of the exposure parameter according to the preset target brightness and exposure statistical data.

[0119] In the sixth possible implementation provided based on the above fourth possible implementation or the above fifth possible implementation, the image processing chip controls the camera to start up quickly based on the target operating parameters, including:

[0120] The image processing chip sets an initialization sequence for configuring the image sensor based on the initial value of the exposure parameter;

[0121] The image processing chip sends the initialization sequence to the image sensor so that the image sensor performs configuration operations based on the initialization sequence and switches to work in the second mode.

[0122] In the seventh possible implementation provided based on the above sixth possible implementation, the processor 402 also implements the following steps when running the computer program stored in the memory 401:

[0123] After the image sensor switches to work in the second mode, the image processing chip performs image processing on the second image output by the image sensor;

[0124] When the processed second image does not meet the preset conditions, the image processing chip discards the processed second image until the processed second image meets the preset conditions, and then the image processing chip starts to output the processed second image to the user.

[0125] In the eighth possible implementation provided based on the above seventh possible implementation, the frame rate of the image sensor in the second mode is initially the specified second frame rate; the processor 402 also implements the following steps when running the computer program stored in the memory 401:

[0126] When the image processing chip starts to output the processed second image to the user, it controls the frame rate of the image sensor to switch from the second frame rate to the specified third frame rate.

[0127] In a ninth possible implementation manner provided based on the above first possible implementation manner, the image convergence statistical information includes: white balance gain statistical data; the target operating parameter includes: the initial value of the white balance gain parameter; the image processing chip determines the target operating parameter related to fast startup according to the image convergence statistical information, including:

[0128] The image processing chip determines the initial value of the white balance gain parameter according to the white balance gain statistical data.

[0129] In a tenth possible implementation manner provided based on the above eighth possible implementation manner, the white balance gain statistical data includes: R pixel gain statistical data and B pixel gain statistical data; the R pixel gain statistical data and the B pixel gain statistical data are respectively stored in corresponding preset registers; the image processing chip determines the initial value of the white balance gain parameter according to the white balance gain statistical data, including:

[0130] In the case where the R pixel gain statistical data or the B pixel gain statistical data has reached the maximum value of the corresponding register, the image processing chip determines the initial value of the white balance gain parameter by means of linear interpolation according to the preset white balance parameter mapping relationship and the white balance gain statistical data;

[0131] In the case where neither the R pixel gain statistical data nor the B pixel gain statistical data has reached the maximum value of the corresponding register, the image processing chip determines the white balance gain statistical data as the initial value of the white balance gain parameter.

[0132] It should be understood that in the embodiments of the present application, the processor 402 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0133] The memory 401 may include a read-only memory and a random access memory, and provide instructions and data to the processor 402. A part or all of the memory 401 may also include a non-volatile random access memory. For example, the memory 401 may also store information about the device category.

[0134] As can be seen from the above, in the embodiments of the present application, the image sensor carried by the camera can support collecting small images at a high frame rate; after the image processing chip and the image sensor are powered on, the image sensor first collects small images at a high frame rate. Since the frame rate is high and the data volume of the small images is small, the small images can quickly converge to the normal brightness in a short time, so that the image processing chip can quickly obtain the image convergence statistical information based on the small images, and thereby determine the target working parameters related to the quick start. In this way, the camera can quickly start according to the target working parameters, so as to output images that meet the quality requirements to the user as soon as possible. In the above process, the camera does not need to carry a photosensitive device, and can achieve the quick start of the camera at a lower cost.

[0135] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above division of each functional unit and module is used as an example for illustration. In practical applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments and will not be elaborated here.

[0136] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0137] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, it can also be completed by a computer program instructing related hardware. The above computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the above computer program includes computer program code, and the above computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The above computer-readable storage medium can include: any entity or device capable of carrying the above computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer-readable memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the above computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0138] The above embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that: The image processing method is applied to a camera, the camera is provided with an image processing chip and an image sensor, wherein the image sensor supports a first mode and a second mode, the first mode captures a first image at a specified first frame rate, the second mode outputs a second image, and the resolution of the first image is smaller than the resolution of the second image; the image processing method comprises: After the image processing chip and the image sensor are powered on, initializing the image processing chip and triggering the image sensor to work in the first mode; The image processing chip obtains image convergence statistical information based on data obtained when the image sensor operates in the first mode within a specified time period; After the specified time period, the image processing chip determines target operating parameters related to fast startup based on the image convergence statistical information; The image processing chip controls the camera to start quickly based on the target working parameters, so that the camera outputs images that meet quality requirements to the user.

2. The image processing method according to claim 1, characterized in that: The image convergence statistical information includes: exposure statistical information; the target operating parameters include: scene brightness; the image processing chip determines the target operating parameters related to the quick start according to the image convergence statistical information, including: The image processing chip calculates the scene brightness according to the exposure statistical data and a preset proportionality coefficient.

3. The image processing method according to claim 2, characterized in that: The camera is also provided with a fill light; the image processing chip controls the camera to start quickly based on the target working parameter, including: When the scene brightness is less than a preset brightness threshold, the fill light is triggered to turn on.

4. The image processing method according to claim 2, wherein: The target operating parameters also include: an initial value of an exposure parameter; the image processing chip determines the target operating parameters related to the quick start according to the image convergence statistical information, and also includes: When the scene brightness is less than a preset brightness threshold, the image processing chip determines an initial value of the exposure parameter according to the scene brightness and a preset brightness-exposure parameter correspondence relationship.

5. The image processing method according to claim 2, characterized in that: The target operating parameters also include: an initial value of an exposure parameter; the image processing chip determines the target operating parameters related to the quick start according to the image convergence statistical information, and also includes: When the scene brightness is greater than a preset brightness threshold, the image processing chip determines an initial value of the exposure parameter according to a preset target brightness and the exposure statistical data.

6. The image processing method according to claim 4 or 5, characterized in that: The image processing chip controls the camera to start quickly based on the target working parameter, including: The image processing chip sets an initialization sequence for configuring the image sensor based on the initial value of the exposure parameter; The image processing chip sends the initialization sequence to the image sensor, so that the image sensor performs configuration operations based on the initialization sequence and switches to operate in the second mode.

7. The image processing method according to claim 6, characterized in that: The image processing method further comprises: After the image sensor is switched to operate in the second mode, the image processing chip performs image processing on the second image output by the image sensor; When the processed second image does not meet the preset condition, the image processing chip discards the processed second image, and when the processed second image meets the preset condition, the image processing chip starts to output the processed second image to the user.

8. The image processing method according to claim 7, characterized in that: The image output frame rate of the image sensor in the second mode is initially a specified second frame rate; and the image processing method further includes: When the image processing chip starts to output the processed second image to the user, the image processing chip controls the image sensor to switch the image output frame rate from the second frame rate to a specified third frame rate.

9. The image processing method according to claim 1, characterized in that: The image convergence statistical information includes: white balance gain statistical data; the target operating parameter includes: an initial value of the white balance gain parameter; the image processing chip determines the target operating parameter related to the quick start according to the image convergence statistical information, including: The image processing chip determines an initial value of the white balance gain parameter according to the white balance gain statistical data.

10. The image processing method according to claim 9, characterized in that: The white balance gain statistical data includes: R pixel gain statistical data and B pixel gain statistical data; the R pixel gain statistical data and the B pixel gain statistical data are respectively stored in corresponding preset registers; the image processing chip determines the initial value of the white balance gain parameter according to the white balance gain statistical data, including: When the R pixel gain statistical data or the B pixel gain statistical data has reached the maximum value of the corresponding register, the image processing chip determines the initial value of the white balance gain parameter by linear interpolation according to a preset white balance parameter mapping relationship and the white balance gain statistical data; When the R pixel gain statistical data or the B pixel gain statistical data does not reach the maximum value of the corresponding register, the image processing chip determines the white balance gain statistical data as the initial value of the white balance gain parameter.

11. A camera, comprising a memory, an image sensor and an image processing chip, wherein the image sensor and the image processing chip are both equipped with a processor, and the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 10 is implemented.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by one or more processors, the method according to any one of claims 1 to 10 is implemented.

Images